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ABSTRACT 


This  project  paper  provides  a  systematic  decomposition  and  industrial  comparison 
of  the  U.S.  defense  decision-support  process  methodologies  of  Portfolio  Analysis  (PA). 
Included  are  current  methods,  tools,  and  models  for  ranking  and  evaluating  strategic 
alternatives  and  options.  The  PA  decision-support  process  analyzed  in  the  study  is  used 
by  the  Department  of  Defense  (DoD)  to  allocate  resources  to  satisfy  national  strategic 
goals.  Our  working  premise  is  that  an  effective  decision-support  process  provides  data 
driven  knowledge  directly  to  relevant  decision  makers  to  meet  all  U.S.  defense  and 
national  strategic  requirements,  including  proper  balance  of  costs,  risks,  and  capabilities 
in  both  routine  operational  and  tactical  battle  space  scenarios. 

To  conduct  this  evaluation  (4)  Program  Management  (PM)  and  2  PA  tools  were 
researched  with  a  view  to  modeling  identifiable  strategic  requirements  juxtaposed  with 
technical,  financial,  and  implementation  factors  to  meet  today’s  strategic  capabilities 
requirements.  What  was  learned  is  that  together  PA  and  PM  tools  used  correctly  provide 
useful  tools  for  decision  makers  at  all  levels  of  government  provided  they  are  used 
correctly.  While  utilizing  the  modeling  programs,  experts  in  strategic  and  program 
analysis  need  to  evaluate  acquisition  programs  to  rate  the  potential  Estimated  Military 
Value  (EMV)  in  meeting  the  strategic  needs  of  the  United  States. 
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EXECUTIVE  SUMMARY 


This  is  a  Portfolio  Management  Analysis  assessment  was  conducted  from  August 
2008  through  May  2009.  The  focus  of  this  Joint  Applied  Project  is  identify  and  assess 
current  Commercial  Off-the-Shelf  (COTS)  Portfolio  Analysis  (PA)  software  products  and 
solutions — with  attention  to  market  positioning,  market  share,  product  features,  and  other 
features.  Two  products  (Risk  Simulator  and  Palisades  @Risk)  were  used  to  develop 
Portfolio  Models.  These  models  were  populated  with  relevant  data,  and  then  run  through 
an  appropriate  number  simulation  iterations  to  assess  candidate  projects  with  respect  to 
risk  and  Expected  Military  Value  (EMV). 

This  document  discusses  Portfolio  Management  Analysis  (PMA)  during  various 
stages  of  project  management  and  system  engineering.  The  goal  for  PMA  is  realized  after 
the  entire  project  design  infrastructure  is  implemented  from  agency  heads  to  managers 
and  the  end  users  instruments  are  provided  for  implementation. 

PMA  requirements  and  scaling  become  secondary  to  the  general  strategic  design 
and  fit  of  each  organization.  Even  where  there  is  a  lack  of  desired  leadership, 
organizations  should  implement  best  practices  and  tools  found  in  successful  Portfolio 
Management  Projects.  This  foundation  enables  organizational  growth  by  providing 
design  tools  to  achieve  scalable  requirements,  operational  readiness  and  the  functions 
needed  to  provide  sound  balance  business  practices.  However,  broad  PA  implementation 
could  more  likely  be  slowed  by  policy,  management  and  funding  constraints,  rather  than 
technical  considerations. 

We  found  that  notwithstanding  a  need  for  some  caution;  PA  tools  are  a  way  of 
enriching  and  equipping  Department  of  Defense  (DoD)  communities  with  a  means  of 
professionally  managing  their  resources.  However,  agencies  and  managers  must  balance 
new  DoD  Acquisition,  Technology  and  Logistics  Life  Cycle  Management  Strategic 
Portfolio  Development  policies,  as  directed  by  Congress.  Working  PA  tools  at  this  level 
requires  policy  that  ensures  proper  budget  and  program  identification  of  existing  or 
legacy,  evolutionary  or  new  development.  The  results  of  this  analysis  will  be 
synthesized,  documented  and  recommended  to  defense  military  organizations,  and 


agency  heads  for  consideration.  The  intent  is  to  identify  approaches  and  tools  to 
incorporate  PMA  net-centric  strategies  to  meet  war  fighter  and  business  operations 
requirements,  while  continuing  to  maintain  current  levels  of  service,  ensuring 
conservation  of  manpower  and  meeting  infrastructure  resource  requirements. 

Key  Recommendations 

1.  Develop  PA  education  programs. 

2.  Develop  a  PA  solutions  and  interoperability  lab. 

3.  Develop  a  PA  Center  of  Excellence  (CoE).  The  authors  feel  Naval  Postgraduate 
School  (NPS)  is  ideally  suited  to  develop  the  curriculum,  lab,  CoE,  and  training 
programs.  In  short,  NPS  is  well  suited  to  provide  a  “PA  Solution  Center”  within 
DoD. 
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I.  U.S.  DEFENSE  PORTFOLIO  ANALYSIS 

A.  PORTFOLIO  ANALYSIS 

This  chapter  defines  and  describes  PA;  where  it  started,  what  it  measures,  how 
other  industries  are  utilizing  it,  why  it  is  important  today,  and  what  the  DoD  is  doing  and 
planning  for  the  future. 

Modem  Portfolio  Theory  was  introduced  by  Harry  Markowitz  with  his  paper 
“Portfolio  Selection,”  which  appeared  in  the  Journal  of  Finance  (1952).  He 
demonstrated  that  a  portfolio  of  individual  securities  composed  of  consistently  good  risk- 
reward  characteristics  (e.g.,  stocks  of  all  rail  companies),  could  well  be  foolish.  He 
detailed  the  mathematics  of  diversification,  which  focused  on  selecting  portfolios  based 
on  their  overall  risk-reward  characteristics.  He  felt  that  investors  should  create  portfolios 
of  dissimilar  securities  rather  than  purchase  and  hold  only  individual  securities  (e.g.,  only 
shares  of  IBM).  Portfolio  theory  provides  a  broad  context  for  understanding  the 
interactions  of  systematic  and  non-systematic  risk  and  reward. 

Non-systematic  risk  (diversifiable)  is  specific  to  a  particular  security  or  sector  so 
that  the  impact  on  a  diversified  portfolio  is  limited.  This  indicates  that  DoD  strategic 
decision  makers  must  not  place  all  their  assets  (securities)  in  a  single  asset  type.  The  risk 
is  that  the  DoD  might,  for  example,  plan  an  air  war  with  hundreds  of  planes  but  fail  to 
provide  sufficient  ground  forces  to  follow  up  and  finish  the  job. 

Systematic  risk  (non-diversifiable)  cannot  be  eliminated  by  having  a  portfolio 
with  many  asset  types.  This  risk  affects  the  entire  economy  (or  a  sector  of  it),  and  is  also 
known  as  market  risk.  This  is  important  in  evaluating  economic  risk  when  deciding 
which  assets  to  include  in  the  defense  strategy.  Risk  has  profoundly  shaped  how 
institutional  portfolios  are  managed  and  has  motivated  the  use  of  passive  investment 
management  techniques. 
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The  mathematics  of  portfolio  theory  is  used  extensively  in  Financial  Risk 
Management  (Financial  Risk  Management,  2008)  and  was  a  theoretical  precursor  for 
today’s  Value-at-Risk  (VaR)  (2009)  measures  (Portfolio  Theory). 

PA  is  the  art  and  science  of  allocating  scarce  resources  to  satisfy  strategic 
objectives,  or  determining  how  to  best  spend  limited  dollars  (Flynn  &  Field,  2006).  PA 
also  provides  tools  for  organizing  and  managing  a  set  of  projects  in  a  portfolio  of  projects 
to  meet  its  goal  (COCP,  2004).  Portfolio  Management  begins  with  an  enterprise-level 
identification  and  definition  of  market  opportunities  and  then  prioritization  of  those 
opportunities  within  resource  constraints  (GAO,  2007,  p.  9).  A  set  of  projects  tracked 
across  the  entire  portfolio  in  a  timely  and  effective  manner  helps  senior  leadership  make 
sound  decisions,  data-based  decisions  supported  by  analysis  of  cost,  schedule  and 
performance  risks.  These  future  projects  will  have  a  National  strategic  impact  as 
situations  and  partners  change.  The  ability  of  senior  leadership  to  adjust  portfolios  to 
meet  defense  needs  now  and  in  the  future  is  critical. 

PA  is  used  by  businesses  to  measure  everything  from  money  to  performance.  In 
the  finance  industry,  it  is  used  to  measure  the  strength  of  a  group  of  investments  to  make 
appropriate  tradeoffs  of  expected  return  on  investment  and  risk.  Using  the  Markowitz 
Efficient  Frontier  (MVO,  2009),  a  ratio  of  the  expected  return  for  each  asset,  the  standard 
deviation  of  each  asset’s  logarithmic  relative  returns  (measure  of  risk),  and  the 
correlation  matrix  between  these  assets,  sets  of  portfolios  with  expected  returns  greater 
than  any  other  with  the  same  or  lesser  risk,  and  lesser  risk  than  any  other  with  the  same  or 
greater  return  could  be  identified  (MVO,  2009).  In  the  Information  Technology  (IT) 
sector,  PA  is  used  to  manage  priorities  for  resource  allocation.  Based  on  limited  resources 
(budget),  which  projects  should  we  keep  while  increasing  profits  and  which  are  failing  to 
perform  and  losing  money?  Whatever  is  being  measured  during  the  analysis,  it  is  a  key 
factor  in  the  success  or  failure  of  the  business.  Companies  commonly  use  Net  Present 
Value  (NPV)  analysis,  which  can  show,  in  today’s  dollars,  the  relative  cash  flow  of 
various  alternatives  over  a  long  period  of  time  (GAO,  2007,  p.  15). 

In  general,  successful  companies  take  a  disciplined  approach  to  prioritizing  needs 
and  initiating  a  balanced  mix  of  executable  development  programs  (GAO,  2007,  p.  7). 
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They  begin  with  an  enterprise  level  approach  to  identifying  market  opportunities  and  then 
prioritize  them  based  on  strategic  goals,  resources  available,  and  risk.  The  market 
opportunities  with  the  greatest  potential  to  succeed  are  included  in  the  portfolio. 

IBM  focuses  on  what  it  calls  “industry  integrated  solutions”,  which  are  portfolios 
structured  around  customers’  and  buyers’  behaviors  and  needs  and  not  on  specific 
product  offerings.  IBM  looks  at  the  customers’  needs  and  the  market  segments  that 
might  provide  for  new  opportunities.  The  “Strike  Zone”  is  the  core  business  for  IBM. 
They  then  look  at  the  “traditional”  opportunities  where  current  customers  might  be 
attracted  to  an  existing  business  which  might  be  enhanced.  Then  they  consider  “pushing 
the  envelope”  opportunities  where  they  might  attract  new  customers  or  create  a  new 
business.  On  the  very  outside  would  be  the  “white  space  opportunity”  or  the  most  risky 
business.  This  is  where  IBM  might  develop  something  new  to  the  industry — based  on 
new  customers’  wants  or  needs.  Based  on  available  resources  and  potential  profitability, 
a  balanced  portfolio  across  all  segments  is  obtained  (GAO,  2007,  p.  10). 

Motorola  targets  market  segments  at  the  enterprise  level  to  ensure  a  balanced  mix 
of  projects  and  resources  is  maintained.  Their  Government  and  Enterprise  Mobility 
Solutions  business  unit  uses  a  70-20-10  formula  where  70%  of  the  projects  and  resources 
are  for  the  core  business,  20%  in  new  markets  with  existing  products,  and  10%  for 
discovering  new  markets  and  products  (GAO,  2007,  p.  1 1). 

So  why  is  PA  important  today?  The  Clinger-Cohen  Act  of  1996  mandates  its  use 
for  all  federal  agencies.  The  GAOs  “Assessing  Risk  and  Returns:  A  Guide  for 
Evaluating  Federal  Agencies’  IT  Investment  Decision-Making,”  Version  1,  (Government 
Accountability  Office,  February  1997,  GAO/AIMD-10.1.13)  requires  that  IT  investments 
apply  Return  on  Investment  (ROI)  measures.  DoD  Directive  8115.01  (Department  of 
Defense  Directive,  2005,  DODD  8115.01),  issued  October  2005,  mandates  the  use  of 
performance  metrics  based  on  outputs,  with  ROI  analysis  required  for  all  current  and 
planned  IT  investments.  DoD  Directive  8115.02  (Department  of  Defense  Instruction, 
2006,  DODI  81 15.02)  implements  policy  and  assigns  responsibilities  for  the  management 
of  DoD  IT  investments  as  portfolios  within  the  DoD  Enterprise,  where  they  defined  a 
portfolio  to  include  outcome  performance  measures  and  an  expected  ROI.  The  DoD  Risk 
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Management  Guidance  Acquisition  guide  book  requires  that  alternatives  to  the  traditional 
cost  estimation  be  considered  because  legacy  cost  models  tend  not  to  adequately  address 
costs  associated  with  information  systems  or  the  risks  associated  with  them  (Mun  & 
Housel,  2006,  p.  1).  The  CJCSI  8410.01,  (Chairman  of  the  Joint  Chiefs  of  Staff 
Instruction,  2007,  CJSC  8410.01),  establishes  policies  and  procedures  for  the  War 
fighting  Mission  Area  Information  Technology  Portfolio  Management  and  net-centric 
data  sharing  processes. 

Over  the  next  several  years,  DoD  plans  to  invest  $1.4  trillion  in  major  weapons 
system  programs.  Continued  failure  to  deliver  weapons  systems  on  time  and  within 
budget  not  only  delays  providing  critical  capabilities  to  the  war  fighter,  but  results  in  less 
funding  for  other  DoD  and  federal  needs  (GAO,  2007,  p.  1).  With  this  level  of  spending 
and  an  upcoming  reduction  in  DoD  obligation,  it  is  important  for  the  DoD  to  spend  its 
money  as  efficiently  as  possible.  This  can  only  be  accomplished  by  better  evaluating  the 
programs/systems  for  risk  before  they  start  being  funded  to  truly  ascertain  their  overall 
value  toward  meeting  the  strategic  goals  of  the  U.S.  These  programs  contain 
considerable  risks  in  the  form  of  cost  overruns,  schedule  delays  and  performance  failures. 

So,  what  is  the  DoD  currently  doing?  The  DoD  is  using  individual  program 
managers  to  manage  specific  programs/systems,  without  regard  to  the  overall  strategic 
goal  of  the  U.S.  Each  program  is  its  own  entity,  with  little  or  no  interaction  with  other 
programs  and  program  managers  are  not  held  responsible  for  minimizing  the  risks 
associated  with  their  particular  programs.  The  DoD’s  service-centric  structure  and 
fragmented  decision-making  processes  are  at  odds  with  the  integrated,  portfolio 
management  approach  used  by  successful  commercial  companies  to  make  enterprise- 
level  investment  decisions  (GAO,  2007,  p.  18). 

In  2004,  the  Defense  Finance  and  Accounting  Service  (DFAS)  implemented 
portfolio  management  in  an  effort  to  help  prioritize  initiatives  and  more  closely  link 
budget  to  agency  strategy,  while  answering  a  presidential  call  for  improving  financial 
management.  In  doing  this,  they  developed  an  approach  which  not  only  governs 
technology  investments  but  includes  all  high-value  initiatives  ($250,000  or  more).  As  a 
decision-making  tool,  Portfolio  Management  requires  essential  data  about  all  initiatives 
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to  be  entered  into  a  central  database  and  requires  those  initiatives  to  be  scored  against 
basic  criteria  and  risk  (decision  analysis).  Portfolio  Management  treats  existing  and  new 
initiatives  as  assets  to  be  managed  instead  of  costs.  The  process  is  dynamic  and  iterative 
so  that  the  Portfolio  reflects  changing  agency  goals  and  priorities.  The  key  to  assessing 
portfolio  effectiveness  is  measuring  the  right  things.  Because  of  the  importance  of 
performance  measures  in  completing  the  portfolio  requirements,  it  is  crucial  for  DFAS  to 
agree  on  the  appropriate  measures  early  in  the  Portfolio  Management  process.  Each 
initiative  receives  a  weighted  score  on  three  dimensions:  Mission,  Financial  and  Benefits, 
and  Risk  (Hubbard,  2004). 

Using  the  Joint  Capabilities  Integration  Development  System  (JCIDS),  needs  are 
identified,  technologies  are  evaluated,  milestone  decisions  are  planned  out  and  then 
program  execution  begins.  The  JCIDS  process  emphasizes  early  attention  to  the  fiscal 
implications  of  newly  identified  needs,  including  the  capability  to  divest  the  Department 
of  lower  priority  or  redundant  capabilities.  Despite  these  provisions,  assessments  of  war 
fighting  needs  continue  to  be  driven  by  the  services  and  to  be  based  on  investment 
decision-making  processes  that  do  not  function  together  to  ensure  that  DoD  pursues 
needs  that  its  resources  can  support  (GAO,  2007,  p.  19).  Another  issue  is  that,  unlike 
commercial  enterprises  which  can  make  program  decisions,  the  services  and  program 
managers  can  not.  The  DoD  has  functional  capability  boards  that  oversee  each  of  eight 
functional  areas.  The  boards  lack  the  authority  to  allocate  resources  and  to  make  or 
enforce  decisions  to  divest  their  capability  area  of  existing  programs  to  pay  for  new  ones 
(GAO,  2007,  p.  21). 

B.  PROJECT  BACKGROUND 

According  to  former  Secretary  of  Defense  (SECDEF),  Donald  Rumsfeld,  “What 
you  measure,  improves.”  In  this  regard,  the  DoD  is  adept  at  measuring  the  cost  and  the 
value  of  a  specific  program  to  fulfill  a  specified  mission.  Trade-off  studies  are  conducted 
and  analyses  of  alternatives  are  performed.  Sometimes,  gap  analyses  are  performed.  But 
are  such  comparisons  made  program  to  program?  Are  funding  decisions  made  with  an 
emphasis  on  leadership  strategy  and  national  objectives?  Portfolio  analysis  is  a 
promising  method  to  improve  DoD  business  practices  by  analyzing  a  portfolio  of  systems 
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as  a  whole,  rather  than  analyzing  individual  acquisition  programs  or  cost.  While  the 
above  statement  is  in  fact  true,  PA  can  be  utilized  as  low  as  the  command  level  to 
evaluate  programs  and  projects  using  real  options,  whether  current  or  future,  used  or  not, 
at  no  current  cost  to  the  command.  These  options  can  then  be  sent  to  the  senior 
commands  for  further  evaluation  and  placement  into  higher  level  options,  all  the  way  to 
the  department/service  level.  Once  at  the  department/service  level,  the  decision  makers 
have  a  better  tool  to  evaluate  these  options  against  the  national  strategic  plans  in  order  to 
make  a  better-informed  decision. 

In  the  U.S.  Military  context,  risk  analysis,  real  options  analysis,  and  portfolio 
optimization  techniques  are  enablers  of  a  new  way  of  approaching  the  problems  of 
estimating  ROI  and  the  risk-value  of  various  strategic  real  options  (Mun  &  Housel,  2006, 
p.  1).  This  analysis  can  be  performed  by  running  various  risk  modeling  programs 
including,  Monte  Carlo  Simulations,  Stochastic  Forecasting,  Real  Options  Analysis 
(ROA),  and  Portfolio  Optimization  using  Knowledge  Value  Added  (KVA).  These 
methodologies  help  in  making  the  best  possible  decisions,  allocating  budgets,  predicting 
outcomes,  creating  portfolios  with  the  highest  strategic  value  and  ROI,  and  so  forth, 
where  the  conditions  surrounding  these  decisions  are  risky  or  uncertain  (Mun  &  Housel 
2006,  p.  2;  Davis,  2003). 

What  are  these  modeling  programs?  Monte  Carlo  methods  are  a  class  of 
computational  algorithms  that  rely  on  repeated  random  sampling  to  compute  their  results. 
Monte  Carlo  methods  are  often  used  when  simulating  physical  and  mathematical 
systems.  Because  of  their  reliance  on  repeated  computation  and  random  or  pseudo¬ 
random  numbers,  Monte  Carlo  methods  are  most  suited  to  calculation  by  a  computer. 
Monte  Carlo  methods  are  useful  for  modeling  phenomena  with  significant  uncertainty  in 
inputs,  such  as  the  calculation  of  risk  in  business.  In  Stochastic  Forecasting,  the  objective 
is  to  minimize  a  given  cost  function  that  depends  on  a  large  number  of  discrete  or 
continuous  variables.  ROA  applies  financial  options  valuation  techniques  to  real  physical 
assets  and  capital  budgeting  decisions  (Campbell  &  Harvey,  2002).  ROA  itself  is  the 
right,  but  not  the  obligation,  to  undertake  some  business  decision;  typically  the  option  to 
make,  or  abandon,  a  capital  investment  (Fintor,  2009).  PA  provides  decision  makers  with 
an  efficient  set  of  portfolios,  based  on  minimizing  risk  subject  to  a  particular  return 
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(Walls,  2004).  Risk  modeling  refers  to  the  use  of  formal  econometric  techniques  to 
determine  the  aggregate  risk  in  a  financial  portfolio.  Risk  modeling  is  one  of  many 
subtasks  within  the  broader  area  of  financial  modeling.  Risk  modeling  uses  a  variety  of 
techniques  including  market  risk,  VaR,  Historical  Simulation,  or  Extreme  Value  Theory 
in  order  to  analyze  a  portfolio  and  make  forecasts  of  the  likely  losses  that  would  be 
incurred  for  a  variety  of  risks.  Such  risks  are  typically  grouped  into  credit  risk,  liquidity 
risk,  interest  rate  risk,  and  operational  risk  categories  (Fintor,  2009). 

As  part  of  the  project  background,  we  must  include  a  short  discussion  about  risk. 
So  what  is  risk?  Risk  is  any  uncertainty  that  affects  a  system  in  an  unknown  fashion  and 
its  ramifications  are  unknown,  but  it  brings  great  fluctuation  in  the  value  and  outcome. 
Risk  has  a  time  horizon,  meaning  that  uncertainty  evolves  over  time,  which  affects 
measurable  future  outcomes  and  scenarios  with  respect  to  a  benchmark  (Mun  &  Housel, 
2006,  p.  33).  The  DoD  does  not  just  budget  for  one  year  at  a  time.  There  are  annual  fully 
funded  programs  and  multiyear  funded  programs.  There  are  programs  that  have  annual 
appropriations,  bi-annual  appropriations,  and  five-year  appropriations.  The  question  then 
is,  how  does  the  DoD  plan  for  these  out  year  budgets  while  accounting  for  the  risk  of 
program  failure,  increased  cost,  delays  in  scheduled  deliveries,  changes  in  strategic 
options  and  goals,  or  even  the  inadequacy  of  technological  maturity?  Every  four  years 
we  elect  a  new  administration,  which  creates  its  own  foreign  policy  and  strategic 
objectives.  As  these  objectives  change,  so  must  the  analysis  of  DoD’s  portfolios  and 
their  ability  to  satisfy  our  national  strategic  objectives.  One  would  hope  that  the 
budgeting  process  was  one  in  which  the  DoD  entered  with  its  eyes  wide  open  and  with  a 
full  understanding  of  the  risks  involved.  So  what  could  the  DoD  be  doing  to  minimize 
risk?  First,  instead  of  looking  at  the  short  term  and  making  decisions  based  on  current 
ROI  or  cost,  they  should  look  toward  the  long  term  investment  risk-return  ratio,  total 
strategic  value,  to  include  returns,  cost,  strategic  options,  (options  available  now  or  later, 
whether  taken  or  not),  as  well  as  risk  (Mun  &  Housel,  2006)  and  reduction  of  program 
duplication  across  services. 

The  Pentagon  is  in  the  midst  of  making  investment  decisions  using  so-called 
“capability  groups”  across  the  entire  military,  as  opposed  to  managing  them  on  a  service- 
specific,  program-specific  manner.  Enhancing  the  “portfolio  management”  initiative  is 
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one  of  the  items  on  SECDEF  England’s  list  of  goals  he  would  like  to  accomplish  by  the 
end  of  this  year.  England  launched  the  initiative  in  2006  to  group  closely  related  military 
systems  from  all  Services  and  DoD  Agencies  and  manage  them  in  groups.  A  similar 
mechanism  has  been  established  for  Command  and  Control  (C2)  systems.  The  goal  is  to 
keep  the  military  services  from  pursuing  redundant  and  pricey  platforms.  In  his  February 
7th  memo,  England  directed  that  four  capability  groups,  tested  during  the  construction  of 
the  Pentagon’s  2009  budget  blueprint,  be  made  permanent.  Under  those  now-official 
groups,  military  programs  from  across  the  DoD  will  be  grouped  into  these  four  categories 
to  help  with  investment  decisions:  C2;  Battle  Space  Awareness;  Network-Centric 
Warfare  (NCW);  and  Logistics  (Bennett,  2008). 

Hopefully,  the  DoD  master  national  strategy  will  help  to  guide  investment 
decisions  based  on  the  entire  national  strategy  and  across  all  services  rather  than  on  the 
needs  or  wants  of  a  single  service,  using  capability  requirements.  This  has  been 
addressed  as  recently  as  September,  2008  with  the  release  of  DoD  Directive  (DODD 
7045.20,  2008),  Capability  Portfolio  Management.  The  nine  capability  areas  outlined  in 
the  DODD  are:  C2,  Battle  Space  Awareness,  NCW,  Logistics,  Building  Partnerships, 
Protection,  Force  Support,  Force  Application,  and  Corporate  Management  and  Support. 
The  latter  five  are  in  the  interim  phase  through  Fiscal  Year  (FY)  2010  Program  Objective 
Memorandum.  The  main  policy  point  is  stated  as  such:  The  DoD  shall  use  capability 
portfolio  management  to  advise  the  Deputy  Secretary  of  Defense  and  the  Heads  of  the 
DoD  Components  on  how  to  optimize  capability  investments  across  the  defense 
enterprise  (both  materiel  and  non-materiel)  and  minimize  risk  in  meeting  the 
Department’s  capability  needs  in  support  of  strategy  (DODD  7045.20,  2008). 

We  have  talked  about  both  PA  and  portfolio  management  a  great  deal  up  to  this 
point.  To  clarify,  this  paper  focuses  on  PA,  but  in  doing  so,  we  must  also  discuss 
portfolio  management.  There  are  similarities,  such  that  in  both  portfolio  (decision) 
analysis  and  portfolio  management  one  must  analyze  and  practice  risk  management.  PA, 
using  the  Markowitz  Efficient  Frontier  (MVO,  2009)  optimization  approach,  provides 
decision  makers  with  an  efficient  set  of  portfolios  based  on  minimizing  risk  subject  to  a 
particular  return. 
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Portfolio  Management,  on  the  other  hand,  provides  guidance  as  to  what  level  of 
risk-taking  is  appropriate.  PA  alone  does  not  provide  managerial  guidance  about  which 
efficient  portfolio  is  best  for  the  organization.  Combining  PA  and  portfolio  management 
can  improve  the  overall  decision  process,  and  could  ultimately  improve  organizational 
performance  (Walls,  2004).  This  paper  focuses  on  the  first  phase,  Analysis,  as  depicted  in 
Figure  1  below. 
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Figure  1.  Analysis  Phase  of  Portfolio  Management  Process 


When  the  Honorable  Richard  Greco,  Jr.,  Assistant  Secretary  of  the  Navy 
(Financial  Management  and  Comptroller),  entered  his  position  in  late  2004,  he  was  asked 
by  the  Vice  Chief  of  Naval  Operations  to  consider  devising  methods  that  could  be  used  to 
analyze  programs  and  better  inform  resource  decision  making  early  in  the  Planning, 
Programming,  Budgeting  and  Execution  process.  This  request  was  the  beginning  of  an 
effort  to  analyze  common  and  best  practices  in  government  and  in  the  private  sector  and 
then  to  use  and  augment  those  techniques  in  a  Department  of  the  Navy  (DoN)  construct. 
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A  Mine  Countermeasures  (MCM)  PA  pilot  was  established  and  fifteen  Subject 
Matter  Experts  (SMEs)  were  allowed  to  score  forty  different  systems  and  six  platforms 
using  a  model  similar  to  that  used  at  U.S.  Special  Operations  Command,  the  Strategy- to- 
Task  Model,  which  is  an  exemplary  DoD  model.  In  the  MCM  pilot,  common  criteria 
were  established  based  on  the  strategic  goals  of  the  organization.  In  scoring  capabilities 
and  risks  of  MCM  systems,  a  logical,  rigorous,  strategy-to-tasks  approach  is  used, 
designed  to  link  individual  assets  such  as  ship’s  sonar,  and  influence-sweep  sleds  to 
broad-based,  macro  national  security  objectives.  By  employing  this  approach,  current  and 
proposed  systems  are  evaluated  in  terms  of  their  contribution  to  the  goals  and  priorities 
set. 


C.  PROJECT  METHODOLOGY 

1.  Problem  Identification 

Dr.  Flynn  has  already  identified  common  problems  with  regard  to  existing  DoD 
models  and  portfolio  management  techniques.  Most  existing  DoD  models  fall  short.  For 
example,  Analysis  of  Alternatives  and  campaign  analyses  are  not  PA.  Other  models  and 
processes  consider  only  a  subset  of  tasks,  costs,  and  capabilities  and  do  not  take  a  holistic 
view  of  the  portfolio.  Other  common  problems  include  (Flynn  &  Field,  2006): 

a.  Portfolios  do  not  reflect  national  strategy, 

b.  Poor  overall  quality  with  regard  to  project  selection, 

c.  Lack  of  focus  on  the  right  projects;  resources  are  wasted  on  the  wrong 

projects,  and 

d.  Multi-year  funding  designed  to  save  money  and  improve  the  defense 

industrial  base  not  being  appropriately  utilized. 

Multi-year  contracts  are  expected  to  achieve  lower  unit  costs  compared  to  annual 
contracts,  through  one  or  more  of  the  following  sources: 

a.  Purchase  of  parts  and  materials  in  Economic  Order  Quantities, 

b.  Improved  production  processes  and  efficiencies, 

c.  Better  utilized  industrial  facilities, 
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d.  Limited  engineering  changes  due  to  design  stability  during  the  multiyear 

period,  and 

e.  Cost  avoidance  by  reducing  the  burden  of  placing  and  administering  annual 

contracts. 

The  use  of  multi-year  contracts  does  not  come  without  risks;  such  as  increased 
costs  to  the  government,  should  the  multi-year  contract  be  changed  or  canceled,  and 
decreased  annual  budget  flexibility  for  the  program  and  across  the  DoD’s  portfolio  of 
weapon  systems  (GAO,  2008,  p.  5). 

Part  of  our  methodology  will  be  to  use  the  integrated  risk  analysis  framework  as 
outlined  by  Dr.  Mun.  The  framework  consists  of  eight  distinct  phases  of  a  successful  and 
comprehensive  risk  analysis  implementation.  The  eight  phases  (Mun  &  Housel,  2006, 
pp.  22,  23)  are: 

1 .  Qualitative  Management  Screening 

2.  Time-series  and  Regression  Forecasting 

3.  Base  Cost  NPV  Analysis 

4.  Monte  Carlo  Simulation 

5.  Real  Options  Problem  Framing 

6.  Real  Options  Modeling  and  Analysis 

7.  Portfolio  and  Resource  Optimization 

8.  Reporting  and  Updating  Analysis 

Using  this  framework,  we  will  develop  a  sample  scenario  which  will  approximate 
a  basic  DoD  program,  with  options  over  a  set  time  period.  Using  this  scenario  we  will 
test  several  COTS  programs  to  evaluate  which  product,  in  fact,  provides  the  best  analysis 
for  use  in  the  DoD  based  on  the  eight  phases  of  the  risk  analysis  framework. 

2.  Systems  Engineering 

Systems  Engineering  is  responsible  for  overseeing  the  COTS/Govemment  Off- 
the-  Shelf  (GOTS)  analysis  and  selection  process  and  developing  appropriate  criteria  for 
assessing  and  evaluating  the  applicability  and  fit  of  existing  components  pertaining  to  the 
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current  development  effort.  Gathering  and  assessment  of  COTS/GOTS  (C/G)  product 
data  may  be  performed  by  specific  domain  engineers  or  SMEs.  There  are  new 
methodologies,  which  are  helping  to  make  the  best  possible  decisions,  allocate  budgets, 
predict  outcomes,  create  portfolios  with  the  highest  strategic  value  and  ROI,  and  so  forth, 
where  the  conditions  surrounding  these  decisions  are  risky  or  uncertain  (Mun  &  Housel, 
2006,  p.  2).  For  this  research  paper,  a  Systems  Engineering  effort  has  been  conducted 
using  a  COTS  life  cycle  model.  The  COTS  life  cycle  model  produces  a  system  primarily 
out  of  one  or  more  available  COTS  products.  This  means  that  the  system  functionality 
being  developed,  the  system  performance  and  the  operational  characteristics  are  primarily 
driven  by  what  is  commercially  available.  Requirements  and  system  objectives  are 
developed  to  understand  the  functionality  and  performance  characteristics,  and  are  then 
used  as  requirements  to  select  a  set  of  COTS  products  that  provide  most  of  the  system 
functionality  and  desired  performance.  Instead  of  relying  purely  on  immediate  ROI  or 
cost,  a  project  strategy,  process  innovation,  or  new  technology,  should  be  evaluated  based 
on  its  total  strategic  value,  including  returns,  cost,  and  strategic  options,  as  well  as  risks 
(Mun  &  Housel,  2006,  p.  2). 

3.  COTS  Integration  Life  Cycle 

The  goal  of  the  COTS  Integration  model  is  to  build  an  end  product  that  is 
primarily  an  integration  of  available  COTS  products.  The  outcome  of  this  model  is  a 
product  solution  that  will  meet  customer/user  requirements.  COTS  products  are  identified 
and  analyzed  for  use  in  the  proposed  solution.  A  main  advantage  of  using  the  COTS 
Integration  model  is  the  speed  at  which  a  product  can  typically  be  rolled  out  to  the 
customer/user.  Also,  by  incorporating  COTS  products  for  certain  functions,  there  is 
usually  a  high  level  of  confidence  that  the  end  product  will  work  according  to 
expectations  of  the  COTS  products.  However,  not  all  requirements  may  be  satisfied  by 
the  solution  and  it  is  important  that  the  customer/user  accepts  the  limitations  that  the 
COTS  product(s)  may  impose.  The  customer  may  need  to  trade  specific  functionality  or 
at  least  be  able  to  be  flexible  with  regard  to  the  solution.  For  example,  a  COTS  product 
may  dictate  a  change  to  a  customer  process  flow,  or  may  require  the  customer  to  provide 
needed  functionality  from  another  source.  The  COTS  Integration  life  cycle  promotes  a  set 
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of  activities  that  supports  a  customer’s  need  for  implementing  an  80/20  rule-type  project 
(i.e.,  to  produce  a  system  with  80%  of  the  desired  functionality  with  20%  of  the  cost 
and/or  schedule). 

To  begin  the  analysis,  capability  requirements  are  developed  in  the  form  of 
product  requirements.  Next,  available  COTS  products  are  identified  and  analyzed  for  use 
in  the  proposed  end  product.  The  capability  requirements  are  then  mapped  to  product 
features.  While  the  process  is  simple,  the  spectrum  of  COTS  PA  products  is  broad,  and 
the  requirements  are  specific  to  DoD.  The  analysis  needs  to  encompass  the  multi¬ 
dimensional  aspects  of  product  feature  comparison  and  total-cost-of  ownership.  The 
work  that  Dr.  Flynn  has  done  in  identifying  attributes  of  PA  has  been  used  as 
requirements  for  PA  product  research  and  comparative  analysis.  Product  literature, 
demonstrations,  downloadable  evaluations,  vendor  interviews,  and  objective  product 
reviews  will  be  used  to  plot  each  PA  solution’s  capabilities  against  the  requirements 
provided  by  Dr.  Flynn  (Flynn  &  Field,  2006). 

4.  COTS  Analysis  and  Selection 

The  C/G  analysis  and  selection  System  Engineering  and  Integration  process 
component  considers  a  wide  range  of  existing  and  available  COTS  and  GOTS  system 
products  (ranging  from  individual  hardware  components  or  software  algorithms  or 
modules  to  large  system  components)  for  use  in  current  system  development.  This 
process  is  based  on  the  technical  assumption  that  certain  parts  of  large  systems  can  be 
effectively  and  efficiently  assembled  through  use  of  existing  components  and  this  use  can 
potentially  reduce  development  costs,  enable  rapid  assembly  and  integration  of  systems, 
and  reduce  the  maintenance  costs  associated  with  the  support  and  future  upgrade  of  the 
overall  system.  There  are  typically  two  major  activities  for  this  process: 

a.  Analysis  of  current  operational  and  support  requirements,  and 

b.  Availability  of  C/G  elements  to  meet  requirements  and  selection  of  the 
appropriate  C/G  products  to  incorporate  into  the  current  system 
development  effort 
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5.  Appropriate  Data 


According  to  Dr.  Flynn,  the  government  needs  to  implement  PA  to  quantify  and 
justify  programming  and  budget  decisions  during  the  Planning,  Programming,  Budget, 
and  Execution  process  (Flynn  &  Field,  2006).  He  asserts  that,  when  armed  with  this 
analysis,  senior  leadership  can  alter  projects,  drop  projects,  or  increase  projects  based  on 
current  threats  and  proposed  future  needs.  Establishing  a  portfolio  of  projects  that  can  be 
altered,  based  on  the  current  world  situation,  best  value  for  the  government  dollar  and 
new  technology,  is  a  best  business  practice  that  other  industries  are  already  utilizing.  Dr. 
Flynn  further  asserts  that  the  private  sector  uses  a  common  metric  (i.e.,  dollars)  to 
determine  value. 

There  is  no  commonly  defined  metric  for  value  across  DoD  programs.  As  a 
result,  military  value  is  extremely  difficult  to  determine  and  must  be  subjective.  Dr. 
Flynn  has  provided  information  on  the  criteria  and  data  that  is  required  to  conduct  PA  for 
the  purpose  of  evaluating  programs  comparatively.  He  proposes  replacing  the  Expected 
Commercial  Value  model  (used  in  commercial  contexts)  with  an  EMV.  The  National 
Security  Strategy  of  March  2005  identifies  four  strategic  objectives  and  eight  required 
operational  capabilities  of  U.S.  military  forces.  The  EMV,  in  one  of  many  constructs,  is  a 
function  of  the  strategic  importance  of  a  project,  the  degree  to  which  the  capability  is 
desired,  as  well  as  probabilities  of  technical  and  operational  success.  It  is  important  to 
note  that  the  variable  for  Strategic  Importance  is  influenced  directly  by  strategic 
objectives;  similarly,  the  degree  of  capability  desired  is  influenced  by  key  operational 
capabilities. 

6.  Life  Cycle  Activities 

Engineering  activities  in  the  COTS  Integration  life  cycle  begin  primarily  with  an 
iterative  interaction  between  the  Requirements  and  Concept  Development,  Analysis  and 
Initial  Design,  Architecture  Definition  and  C/G  Analysis  and  Selection  components. 
Within  Requirements  and  Concept  Development  a  set  of  initial  system  objectives  are 
elicited  and  compiled  into  a  product  vision  or  feature  list.  This  information  is  then  used 
to  identify  and  collect  information  about  existing  products  that  could  satisfy  customer 
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needs.  Trade  Studies  represent  a  key  technical  management  component  that  supports  the 
life  cycle  activities.  Among  the  greatest  risks  in  a  C/G-type  project  are  the  actual 
performance  of  the  selected  products  and  the  performance  of  C/G  products  that  may  be 
integrated  and  interfaced  in  ways  that  were  previously  not  considered.  Trade  Studies, 
conducted  within  the  Decision  Analysis  and  Resolution  component,  using  a  set  of 
potential  C/G  products  executing  in  realistic  scenarios,  can  significantly  reduce  this  risk 
to  the  project. 

7.  Design,  Integration  and  Evaluation  Activities 

Once  the  prospective  COTS  products  have  been  identified,  Detailed  Design  and 
Verification  activities  can  commence.  Detailed  Design  is  focused  on  the  system 
capabilities  that  are  needed  to  enable  the  selected  COTS  products  to  interact  with  each 
other  and/or  the  user  (e.g.,  glue  components).  Therefore,  it  should  represent  a  very 

minimal  activity.  It  is  conceivable  that  Detailed  Design  may  not  even  be  needed  (e.g.,  a 

single  COTS  system  is  selected  to  provide  the  entire  functionality).  Early  Verification 
activities  focus  on  demonstrating  that  the  chosen  C/G  products  actually  perform  as 
anticipated  on  the  project  and  support  the  required  interfaces.  Verification  of  the 
components  can  be  thought  of  as  “unit  tests”  for  the  components  that  are  going  to  be 
integrated  into  the  delivered  system. 

8.  System  Engineering  Activity  Descriptions 

a.  Perform  Initial  Analysis  of  C/G  Elements 

Our  research  team  will  perform  initial  analysis  of  the  C/G  elements, 
including  identifying  the  screening  criteria  for  evaluation  of  GOTS/COTS  candidates  and 
evaluating  the  properties  of  candidate  C/G  components.  Such  properties  include 
component  functionality  (what  services  are  provided)  and  other  aspects  of  a  component’s 
interfaces  (e.g.,  standards  implemented,  operational  environment).  These  properties  also 
include  quality  and  supportability  aspects  of  the  components  that  may  be  more  difficult  to 
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isolate  and  quantify  than  component  functional  and  performance  aspects.  Based  on  the 
screening,  a  short  list  of  candidate  C/G  components  will  be  identified  for  use  in  the  next 
activity. 


b.  Perform  Literature  Review 

Our  research  team  will  collect  information  on  PA  solutions  to  obtain  case 
studies,  and  industry  best  practices.  The  following  sources  will  be  used: 

•  DoD  components  and  defense  firms 

•  Journal  articles 

•  Academic  and  Wall  Street  SMEs 

•  Periodicals  and  magazines  which  conduct  product  comparisons 

•  Other  industries  utilizing  PA 

c.  Select  Appropriate  C/G  Elements  for  Inclusion  in  Current 
System  Development 

Our  research  team  will  determine  the  “fitness  for  use”  of  existing  C/G 
components  that  are  candidates  for  application  in  the  new  system  development  effort. 
We  will  then  determine  which  available  C/G  components  represent  a  “best  fit”  for  the 
current  development  effort.  It  is  also  reasonable  to  discover  “non-technical”  component 
properties,  such  as  the  vendor’s  market  share,  past  business  performance,  and  process 
maturity  of  the  developing  organization.  The  selection  process  may  also  extend  to 
include  qualification  of  the  development  process  used  to  create  and  maintain  candidate 
C/G  products.  The  determination  on  use  of  existing  C/G  products  may  be  dictated  by  the 
acquirer  or  another  appropriate  stakeholder.  Thus,  this  process  may  actually  be  executed 
in  advance  of  completion  of  the  Requirements  and  Concept  Development  process  since 
the  outcome  of  the  C/G  analysis  and  selection  may  impact  the  development  and 
definition  of  system  requirements. 
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II.  SYSTEM  ENGINEERING  (SE) 


A.  REQUIREMENTS 

1.  Objectives 

SE  begins  with  the  capture  of  requirements.  For  this  research  project,  several 
sources  were  used  to  establish  requirements.  In  understanding  the  requirements  provided 
from  different  organizations,  it  is  necessary  to  understand  how  each  organization  defines 
PA.  The  Project  Management  Institute  defines: 

Portfolio  as  “A  collection  of  projects  and  programs  and  other  work  that  are 
grouped  together  to  facilitate  effective  management  of  that  work  to  meet  strategic 
business  objectives.  The  projects  or  programs  of  the  portfolio  may  not  necessarily  be 
interdependent  or  directly  related.” 

Portfolio  Management  (PfM)  as  “The  centralized  management  of  one  or  more 
portfolios,  which  includes  identifying,  prioritizing,  authorizing,  managing,  and 
controlling  projects,  programs  and  other  related  work  to  achieve  specific  strategic 
business  objectives.” 

PfM,  in  the  context  of  this  research  paper,  views  each  program  investment  from 
an  enterprise  level  as  contributing  to  the  collective  whole,  rather  than  an  independent  and 
unrelated  program  investment.  With  this  enterprise  perspective,  Portfolio  Managers  can 
effectively: 

a.  Identify  and  prioritize  opportunities  and 

b.  Apply  available  resources  to  potential  products  to  select  the  best  mix  of 
products  to  exploit  the  highest-priority  and/or  most  promising — 
opportunities  (GAO,  2007) 

This  type  of  approach  depends  on  strong  enterprise  governance  with  committed 
leadership;  it  also  depends  on  information  tools  which  provide  the  ability  to  visualize 
complex  data  relationships  in  a  comprehensible  manner. 
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As  depicted  in  Figure  2,  a  PfM  approach  begins  with  an  enterprise-level 
framework  and  definition  of  market  opportunities  and  then  the  prioritization  of  those 
opportunities  within  resource  constraints.  At  each  review  gate,  programs  are  assessed 
against  available  resources,  established  criteria,  competing  programs,  and  the  goals  and 
objectives  of  the  DoD  as  a  whole.  As  alternatives  pass  through  each  review  gate,  the 
number  is  expected  to  decrease,  until  only  those  alternatives  with  the  greatest  potential  to 
succeed  make  it  into  the  product  portfolio  (GAO,  2007). 


Figure  2.  PfM  Approach  to  Product  Investments  (From  GAO,  2007) 


In  support  of  the  framework  identified  as  a  best  practice  by  the  GAO  (GAO, 
2007),  a  fundamental  premise  of  this  paper  is  the  need  to  identify  COTS  systems  that  can 
provide  the  capabilities  identified  in  the  Requirements  section. 

This  project  analyzes  applicable  PA  tools,  systems  and  underlying  methodologies 
in  terms  of  relevancy  to  identification  of  viable  requirements  and  technical  alternatives. 
As  depicted  in  Figure  3,  the  PA  activities  include: 

•  Identifying  and  Maintaining  the  Baseline 
o  Collecting  data 

o  Grouping  Investments  into  Capability  Areas 
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•  Identifying  Gaps 

o  Using  the  formula  GAPS  =  OBJECTIVES-BASELINE 

•  Identifying  Initiatives  to  Close  Gaps 


Analysis 


Objectives  -  Baseline  =  Gaps 
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Identify  &  Maintain  Baseline 

►  Bin  investments  into  capability  areas 

►  Colled  and  review  data  on  Baseline 
to  indude  description,  costs, 
schedules,  and  performance 


Identify  Gaps 

>  Compare  visio  objectives  to 

Baseline  -  determine  capability  gaps 


Identify  Initiatives 
to  Close  Gaps 

Develop  Initiatives  to  close  gaps 
(materiel  and  non-materiel) 


Figure  3.  Portfolio  Analysis  (PA)  Activities  (From  Mean-Variance  Optimization,  2009) 


2.  Scope 

This  research  project  encompasses  PA  pertaining  to  the  DoD,  as  identified  in  the 
requirements  provided,  both  explicit  and  derived.  Some  interpretation  of  requirements  is 
needed  as  requirements  are  decomposed  into  functional  capabilities.  These 
interpretations,  in  terms  of  this  paper,  have  not  been  presented  to,  nor  vetted  with  the 
authors  and  agencies  that  provided  the  top-level  requirements.  The  research  scope  fully 
explores  the  applicability  of  PA  systems,  vis-a-vis  the  requirements,  using  a  System 
Engineering  approach. 
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This  report  focuses  on  identifying  current  COTS  PA  systems,  operational 
constraints  of  current  systems,  and  general  recommendations  for  system 
implementations.  The  technology  evaluated  includes  PA  software,  computers  and  related 
technology.  The  approach  includes  gathering  the  information  needed  (requirements,  best 
practices)  to  conduct  COTS  product  evaluations. 

3.  Identified  Needs  and  Best  Practices 

This  section  begins  to  address  the  research  questions  in  terms  of  performance, 
functional  requirements,  and  system  requirements.  As  outlined  in  the  Project 
Methodology,  a  Literature  Review  was  conducted  to  examine  the  current  PA  landscape 
from  a  requirements  perspective.  The  key  requirements  were  drawn  from  the  sources 
shown  in  Appendix  B. 

4.  Capability  Requirements 

Table  1  depicts  the  Capability  Portfolio  Analysis  as  a  function  of  Cost,  Schedule, 
and  Performance. 


Table  1.  Capability  Portfolio  Analysis  as  a  Function  of  Cost,  Schedule  and  Performance 


Portfolio  Item 

Capability 

Development 

Capability 

Equipment 

Components 

Capability 

Interfaces 

Capability 

Installation 

Capability 

Life-Cycle 

Logistics 

Notional 

Baseline 

Current 

Available 

Capability 

Current 

Component 

Availability 

Current  System 
or  Equipment 
Interoperability 

Historical 

Installation 

Cost  and 
Schedule  for 
similar 
Capability 

Historical 
administration, 
training,  parts 
kits, 

publications, 
design  changes, 
etc.  for  similar 
Capability 

Cost  as  a 
Function  of 
Increase, 
Decrease,  or 
Change  to  any 
Portion  of  the 
Program 

Changes  to 
application  of 
the  Capability 
(all  changes 
equate  to  cost 
increase  and 
schedule  impact 
+/-) 

Cost  will  be 
constant  or  an 
increase  for 
systems  if  any 
components, 
software,  etc. 
are  changed. 

Cost  increase 
may  be  offset 
by  reduction  in 
components. 
May  also  affect 

Cost  may  be 
same  as  notional 
or  will  be  +  for 
integration  with 
any  new 
components. 
Equates  to 
information 
distribution, 
whether  internal 
to  the  equipment 
or  as  external 

Cost  may  be 
same  as 
notional  cost, 
an  increase,  or 
a  decrease 
depending  upon 
a  number  of 
factors, 
including 
location, 
difficulty  of 
installation  or 

Only  historical 
baseline  costs 
will  remain  the 
same.  Any 
change, 
regardless  of 
increase  or 
decrease  in 
capability  will 
equate  to  a 
positive  cost 
over  baseline. 
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Portfolio  Item 

Capability 

Development 

Capability 

Equipment 

Components 

Capability 

Interfaces 

Capability 

Installation 

Capability 

Life-Cycle 

Logistics 

schedule  and/or 
performance. 

interfaces. 

delivery,  and 
differentials  to 
security, 
environment 
and  others. 

Schedule  as  a 
Function  of 
Weighted  Tasks 
and  Earned 
Value  Metrics 

Any  changes  to 
baseline 
capability  will 
have  an  impact 
to  schedule,  but 
established 
delivery  date 
must  be  the 
goal. 

Whether  COTS 
or  GOTS 
based, 
component 
availability  that 
satisfies  the 
capability 
affect  schedule 
+/-. 

Level  of 
integration  of 
components  of 
the  capability 
affects  tasking 
and  Earned 

Value  within  the 
schedule  +/-. 

Degree  of 
difficulty, 
location,  etc. 
will  affect 
schedule  +/-. 

Procurement 

and 

Documentation 
approval  may  or 
may  not  affect 
overall  schedule 
+/-. 

Performance  as 
a  Function  of 
Technology 
Readiness 

Levels  (TRL) 
(1-9)  and 
Manufacturing 
Readiness 

Levels  (MRL) 
(1-10) 

Assume  legacy 
capability  as 
constant  (TRL 

9  and  MRL  10). 
Any  change  to 
capability  will 
have 

commensurate 
impact  on 
Technology  or 
Manufacturing 
Readiness 
either  +/-. 

Legacy 
components, 
whether  COTS 
or  GOTS  are 
TRL/MRL  9/10 
respectively. 
Current  state  of 
portfolio 
components 
will  impact 
cost,  schedule, 
and 

performance. 

Higher 

TRL/MRL  the 
less  impact  on 
cost  and 
schedule,  but 
may  impact  life 
cycle  logistics, 
interfaces,  and 
installation 
significantly. 

COTS  vs.  GOTS 
are  significant 
factors  for 
interfacing  with 
other  systems. 
Even  with  high 
TRL/MRL 
degree  of 
difficulty  may  be 
deciding  factor. 
COTS  may  be 
acceptable  as 
standalone 
interim  solution 
with  a  migration 
plan  for  later 
GOTS 
integration. 

Installation 

Phase  assumes 
high 

MRL/TRL,  but 
physical  or 
electronic 
characteristics 
must  be 
considered  in 
performance  as 
these  may 
impact  cost  and 
schedule. 

Affected  +/- 
depending  on 
degree  of 
acquisition 
management 
interfaces 
required  and 
Performance 
Based  Logistics 
level  of  effort 
for  life  cycle 
sustainment. 

The  Cost  of  Capability  Concept  must  also  be  considered,  but  may  be  viewed  as  a 
pre-expenditure,  or  constant  plus  a  fixed  fee  for  changes.  The  capability  concept 
document  is  pre-portfolio  selection,  and  assembly  of  capabilities  as  portfolios  to  meet  the 
mission  need  should  be  based  on  this  concept  document.  Inflation  over  time  is  a  constant 
for  any  capability  selection  from  the  portfolios  and  is  not  considered  a  major  factor  in 
selection. 
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5. 


Functional  Requirements 


The  Requirements  Matrix,  Table  2,  shows  the  payload  of  the  SE  requirements 
analysis.  The  requirements  matrix  provides  the  foundation  for  analyzing  the  products 
vis-a-vis  the  requirements.  The  requirements  are  specified  using  performance 
characteristics  which  are  decomposed  to  include  functional  aspects,  system  aspects,  and 
identification  of  tests  that  are  used  to  ensure  the  requirements  are  satisfied. 


Table  2.  Requirements  Matrix 


Category 

Classes 

Measures 

Metrics 

Group  By 

Notes 

Operations 

Mission 

Effectiveness 

Speed  of 
command 

Time 

Capability, 

Service 

Fact 

System 

Availability 

Ao 

System 

Fact 

Quality 

Morbidity 

Facts, 

Score 

User  Defined 

[numeric  value] 

Score 

Public  Service 

0/1  (Binary) 

Score 

Transformational 

Impact 

0/1  (Binary) 

Score 

Situational 

Awareness 

#of  Tracks 

Fact 

Interoperability 

Military 

Standard 

Score 

Category 

Classes 

Measures 

Metrics 

Group  By 

Notes 

Logistics 

Footprint 

Lay  down  Effects 

Space, 

Inventory 

Quantity, 

Training 

Material, 

Non-Material 

Facts, 

Scores 

Acquisition  Risk 

Item  Value 

Unit  Cost 

Fact 

Supplier 

Leverage 

Number  of 
Suppliers 

#Qualified 

Suppliers 

Fact 

Total  Cost  of 
Ownership 

Life  cycle  Cost 

Complete 

System  Costs 

Fact 

Category 

Classes 

Measures 

Metrics 

Group  By 

Notes 

Strategic 

Alignment 

Business  Model 

Organizational  Fit 

0/1  (Binary) 

Strategies, 
Term  (short, 
long) 

Score 

Organizational 

Impact 

Training 

Cost 

Fact 

Personnel 

#People,  #Man 
Hours 

Facts 

Business  Process 
Improvement 

Scope 

ROI/EMV** 

Fact 
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Category 

Classes 

Measures 

Metrics 

Group  By 

Notes 

Cost/Benefit 

Dollar  Value 

Return  On 
Investment 

Cost,  Payback, 
Payback  Period 

Phase 

(RDT&E, 

Production, 

O&S) 

Fact 

EMV 

Return  On 
Investment 

Cost,  EMV, 

EMV  Payback 
Period 

Fact 

Category 

Classes 

Measures 

Metrics 

Group  By 

Notes 

Execution 

Performance 

Schedule 

EVM 

Schedule 

Value 

Program, 

Service, 

Phase 

Fact 

Cost 

EVM 

Cost  Value 

Fact 

Category 

Classes 

Measures 

Metrics 

Group  By 

Notes 

Risk 

Probability 

Rating 

1  through  5 

Ratings, 

Percentage 

Program, 

Service, 

Phase 

Score 

Consequence 

1  through  5 

Ratings, 

Percentage 

Score 

Category 

Classes 

Measures 

Metrics 

Group  By 

Notes 

Organizational 

Impact 

Training 

People  Affected 

Count 

Program, 
Service,  Phase 

Fact 

Personnel 

People  Affected 

Count 

Fact 

Quality  of  Life 

Surveys 

Survey  Metrics 

Score 

**  The  intent  of  EMV  is  to  create  a  single  metrics  equivalent  to  dollars  in  commercial  PA. 


B.  ANALYSIS 

1.  Portfolio  Capability  Evaluation 

From  the  DoD  acquisition  perspective,  capabilities  to  meet  identified 
requirements  that  satisfy  mission  gaps  in  execution  of  military  strategies  are  tied  to  cost, 
schedule,  and  performance.  In  turn  then,  it  is  logical  to  view  portfolios  in  a  manner  that 
optimizes  these  values.  Let  us  consider  that,  for  an  optimum  identified  capability 
requirement,  the  following  is  true,  regardless  of  whether  it  is  from  existing  legacy, 
evolutionary,  or  new  development  (Figure  4): 


23 


operating 

Capabitity+ 

Capability  = 

Capability 

concept 

Capability— 

Capability  Capability  Capability 

Concept  Development  Equipmen 

Capability  Capability  Capability 

t  Interfaces  Installation  Logistics 

Figure  4.  Capabilities  Equation 


The  summation  of  sub-folders  in  a  notional  capability  portfolio  will  result  in  a 
final  capability  that  matches  the  DoD  mission  gap  requirement,  is  better  than  the 
requirement,  or  is  somewhat  less  than  the  requirement,  but  is  good  enough  to  serve  as  an 
interim  solution  (a  lower  threshold  must  be  established  to  know  when  a  sub-folder  must 
be  discarded  as  not  useful).  In  turn,  each  of  the  pieces  of  this  notional  capability  (let  us 
call  them  $K  or  notional  constant  dollars  for  the  baseline)  portfolio  may  be  further 
decomposed  such  that  (Figure  5): 


$K  cc 

+  $Kcd 

+A>  +  $K  L  +  *Kc+'+A  +*Kci;'+A  + 

*K  ,*>*  ">9  A 

Where: 

$K 

Is  cost  of  capability  concept 

“Sd 

Is  cost  of  capability  development 

o 

CD 

Is  cost  of  capability  equipment/components 

*K  cif 

Is  cost  of  capability  interfaces 

$K 

cm 

Is  cost  of  capability  installation 

tK  log 

Is  cost  lifecycle  logistics  forthe  capability  including  contract,  initial  administration,  training  (T), 
publications  (P),  spare  parts  kits  (Spk),  design  changes  (Sys)  forthe  installation,  software 
packages,  etc. 

TA  +  PA+  EpkA+  =  log  A 

Figure  5.  Summation  of  Capabilities 


As  shown  above,  changes  (or  the  +/-  deltas)  to  the  notional  baseline  capability 
result  in  increases/decreases  for  cost,  schedule  and/or  performance.  At  the  enterprise 
level,  then,  these  parameters  may  be  used  to  graphically  show  advantages  and 
disadvantages  for  various  options  within  each  of  the  portfolio  sub-folders  (we  will 
examine  the  individual  parameters  graphically  later  on).  This  same  rationale  may  be  used 
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for  evaluating  schedule  and  performance.  When  numerical  values  are  assigned, 
portfolios  of  capabilities  and  their  components  may  be  evaluated  to  select  those  most 
favorable  within  cost,  schedule  and  performance  desired  to  meet  the  capability  gap.  Note 
that  risk  is  not  an  issue  in  capability  parameter  selection  at  this  point.  Risk  is  an  integral 
part  of  sub-folder  or  technology  evaluation  that  may  make  a  specific  capability  option 
within  the  portfolio  infeasible.  Risk  may  be  evaluated  at  each  enterprise  level  calculation 
and  at  each  subsequent  parent-child  decomposition,  such  that  risk  is  always  a 
consideration  throughout  the  selection  process. 

Taken  in  aggregate,  a  graphical  representation  of  notional  capability  fielding  may 
be  presented,  as  depicted  in  Figure  6.  Once  analyses  of  different  aspects  of  the  portfolio 
(the  parametric  deltas  from  notional  baseline)  are  examined,  the  graphical  representation 
of  the  actual  effort  may  be  very  different.  Minimum  and  maximum  arbitrary  thresholds 
will  determine  whether  the  portfolio  sub-folder  capability  option  is  able  to  meet  cost, 
schedule,  and  performance  within  the  timeframe  desired  by  the  tasking  activity.  If  yes, 
then  the  portfolio  sub-folder  should  be  examined  further.  If  no,  then  the  capability  option 
should  be  tabled  for  future  consideration,  or  as  a  “fallback”  option. 


Figure  6.  Graphical  Representation  of  Notional  Capability  Fielding 
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In  measuring  the  parametric  deltas/differences  for  a  cost  valuation,  consider  a 
matrix  similar  to  Figure  7,  where  notional  cost  is  shown  as  a  function  of  historical 


spending  rates  for  capability  development  and  implementation: 


Total  percent 

2% 

4% 

4% 

3% 

3% 

7% 

10% 

11% 

11% 

13% 

14% 

18% 

100% 

Concept  IK 

0.1% 

Capability 

Development 

10.0% 

2.1% 

2.1% 

0.7% 

1.0% 

2.9% 

2.9% 

2.1% 

2.1% 

0.7% 

0.7% 

0.7% 

28.0% 

Capability 
Equipment  & 
Components 

0.0% 

0.7% 

0.7% 

0.7% 

1.0% 

2.8% 

5.7% 

6.4% 

6.4% 

7.9% 

9.3% 

10.0% 

51.6% 

Capability 

Interfaces 

0.3% 

0.7% 

0.7% 

0.7% 

0.3% 

0.4% 

0.4% 

0.7% 

0.7% 

2.1% 

2.1% 

3.0% 

12.1% 

Capability 

Installation 

0.0% 

0.0% 

0.0% 

0.7% 

0.3% 

0.3% 

0.3% 

0.7% 

0.7% 

0.7% 

0.7% 

3.0% 

7.4% 

Capability 

Lifecycle  0.9%  07%  0.7%  0.0%  0.3%  0.7%  0.7%  1.4%  1.4%  1.4%  0.7%  1.4%  10.3% 

Logistics 

Figure  7.  Notional  Cost  as  a  Function  of  Historical  Spending  Rates 


Adding  or  subtracting  different  parameters  desired  from  the  capability  portfolio 
sub-folders  gives  realistic  notional  thresholds  for  where  expected  costs  will  fall 


throughout  the  capability  development  and  fielding,  as  depicted  in  Figure  8. 


Figure  8.  Realistic  Notional  Cost  Thresholds 
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Evaluation  of  the  portfolio  must  also  consider  the  desired  timeframe  for  fielding 
the  capability.  If  technologies  or  equipment  within  the  selected  sub-folders  have  low 
probability  of  reaching  the  fielding  date,  they  must  be  shelved  until  they  are  mature 
enough  for  consideration.  Schedules  for  the  selected  comparison  parameters  must  use 
arbitrary  weighted  earned  values.  Depending  upon  the  capability  requirement,  there  are 
several  types  of  scheduling  software  products  that  may  be  used.  Regardless  of  the 
scheduling  tool,  as  long  as  it  is  consistent,  a  reasonable  comparison  may  be  made 
between  portfolio  sub-folders  that  meet  notional  Earned  Value  Management  System  and 
Integrated  Defense  Acquisition,  Technology,  and  Logistics  management  framework 
milestone  and  alternative  decision  point  requirements. 


FYXX 

FYXX 

FYXX  |  FYXX 

FYXX  1  FYXX 

Q1  13 

Nil  tfe  Q3  Qd  I  <3-1  Q2  ©  Qd  jVl 

; I; Im; j; jjiwiaf; j; jw  jiiasjijaKjiirsw  , 

capability 

concept 


l 


Implementation 


tieiopiimeint  y  ^ ^ r~ T 


capability  interfaces 

Functional 
Requirements/ 


IP 

System  Design  " 

1  ill  p  i  sm  wnCau  on/ ■  i  i  r 

r 

Platform-Detail 
Desicjn  /  System 


capability  equipment/components 


lifecycle  logistics  for  the  capability  must  keep  pace  with  development,  contracting, 
and  Testing 


Figure  9.  Simple  Capability  Portfolio 


Very  simple  capability  portfolios  may  only  require  something  as  elementary  as 
Figure  9  above  to  select  the  best  options  within  the  sub-folders.  Other  complex 
capabilities  may  require  in-depth  project  breakdown,  and  individual  parameter 
comparisons  within  established  thresholds,  as  shown  in  Figure  10.  Sub-folders  may  also 
have  spin-out  increments  to  fall  back  on  legacy  technologies,  should  risk  mitigation  be 
insufficient  to  continue  a  specific  variable. 
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Figure  10.  Complex  Capability  Portfolio 


Evaluation  of  possible  capability  variances  may  be  set  down  in  simplistic 


spreadsheets  as  depicted  in  Figure  1 1  below: 


SCHEDULE 

as  a  function 
ofweighted 
Earned  Value 
Tasking 

01 

Q2 

Q3 

Q4 

Q5 

Q6 

Q7 

QB 

Q9 

Q10 

Q 1 1 

Q 1 2 

JCIDS 

MS  A 

MS  B 

MS  C 

LRJP 

FRP 

Capability 

Development 

Option  A 

Option  B 

Capability 
Equipment  & 
Components 

spinout 
option  A 

spinout 
option  B 

Capability 

Interfaces 

option  A 

option  B 

Capability 

Installation 

Capability 

Lifecycle 

Logistics 

PBL 

Figure  11.  Evaluation  of  Possible  Variances  Depicted  in  Spreadsheet 


When  overlaid  upon  the  notional  schedule  with  scheduling  thresholds,  as  shown 
in  Figure  12,  one  can  readily  see  where  spinouts  might  be  implemented  across  the 
capability  fielding  process.  This  process  is  especially  useful  in  “design  to  price”  or 
“turnkey”  delivery  of  capabilities. 
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Figure  12.  Notional  Schedule  with  Scheduling  Thresholds 


Although  the  capability  performance  may  be  measured  by  several  criteria,  it  may 
be  best  measured  via  established  criteria,  as  set  forth  in  a  Technology  Readiness 
Assessment  and/or  Manufacturing  Readiness  Assessment.  These  criteria  are  well- 
recognized  throughout  the  DoD  and  other  government  activities.  Because  of  the 
expandable  nature  of  the  requirements  for  each  level  of  performance,  each  of  the 
parameters  for  evaluating  a  sub-folder  within  the  portfolio  may  be  tailored  to  fit  the 
criteria  of  technology  and  manufacturing  readiness.  When  compared  with  one  another, 
using  the  same  criteria,  selection  of  the  most  promising  sub-folders  for  the  capability  can 
be  accomplished. 

Because  sub-folders  consider  existing,  evolutionary,  and  new  developments,  use 
of  this  method  allows  the  evaluator  to  be  able  to  inject  capabilities  into  the  performance 
versus  time  chart  at  the  current  level  and  select  those  that  may  be  more  mature  and, 
therefore,  have  the  best  chance  of  success,  all  other  weighted  factors  being  near  equal. 
The  evaluator  must  be  attuned  to  the  pitfalls  of  selecting  mature  technologies,  even 
though  they  meet  a  current  need,  which  cannot  evolve  and  remain  interoperable  with 
other  capabilities  in  the  battle  space  in  the  out  years.  A  simplistic  summary  of 
technology  and  manufacturing  readiness  and  assessment  criteria  is  shown  in  Figure  13. 
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Figure  13.  Summary  of  Technology  &  Manufacturing  Readiness 

and  Assessment  Criteria 


In  evolutionary  and  developmental  sub-folders,  it  must  be  noted  that  the  notional 
performance  of  the  capability  must  be  at  a  TRL  of  6  by  Milestone  B,  as  set  forth  in  Title 
10  (see  Figure  14). 


Figure  14.  Capability  at  Technology  Readiness  Level  6  by  Milestone  B 
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As  with  scheduling,  a  simplistic  spreadsheet,  as  depicted  in  Figure  15  below,  may 
be  completed  for  the  notional  capability  and  then  parameters  from  the  sub-folders 
overlaid  to  match  projected  performance  growth  over  the  notional  timeline  for  the 
required  fielding  date  of  the  capability.  Again,  there  are  many  software  tools,  both 


simple  and  complicated,  that  may  be  used  to  perform  this  function. 


PERFORMANCE 

as  a  Function  of 
Technology  £L 
Manufacturing 
Readiness 

G1 

G2 

G3 

G4 

G5 

G6 

G7 

GS 

G9 

G1  □ 

G1  1 

G12 

JCIDS 

MS  A 

MS  B 

MS  C 

LRIP 

FRP 

Capability 

Development 

6  or  > 

7  or  > 

A 

0 

OO 

9 

Capability 

Equipment 

Components 

6  or  > 

7  or  > 

A 

0 

00 

9 

Capability 

Interfaces 

6  or  > 

7  or  > 

8  or  > 

9 

Capability 

Installation 

6  or  > 

7  or  > 

B  or  > 

9 

Capability 

Lifecycle 

Logistics 

6  or  > 

7  or  > 

B  or  > 

9 

Notional  TRL  of 
Capability  overall 

6  or  > 

7  or  > 

B  or  > 

9 

Notional  MRL  of 
Capability  overall 

6  or  > 

8  or  > 

9  or  > 

10 

Figure  15.  Simplistic  Spreadsheet  Showing  Notional  Performance  Capability 


Notional  capability  parameters  may  vary,  depending  upon  the  type  of  capability 
being  fielded,  and  the  readiness  level  of  the  parameters  may  vary  widely,  depending  upon 
the  maturity  of  selected  sub-folders  to  meet  the  capability.  However,  as  seen  in  Figure  15 
above,  to  meet  fielding  criteria  there  are  accepted  readiness  levels  that  are  expected  at 
certain  proposed  decision  points  and  milestones  for  the  capability. 

2.  Threat  Environments  (Based  on  the  2008  National  Defense  Strategy 
(NDS) 

a.  Threat  Environment  Current 

Today’s  current  environmental  threat  is:  Global  struggle  against  violent 
extremist  ideology  seeking  to  overturn  the  international  state  system, 
asymmetrical/irregular  warfare. 
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b.  Threat  Environment  Future 

Irregular  challenges,  rogue  states  quest  for  nuclear  weapons  (Iran  and 
North  Korea),  rising  military  power  of  other  states  (China,  potentially  Russia). 

c.  Objectives 

The  objectives  are  as  follows: 

•  Defend  the  Homeland 

•  Win  the  Long  War 

•  Promote  Security 

•  Deter  Conflict 

•  Win  our  Nation’s  Wars 

d.  Achieving  the  Objectives 

Objectives  are  met  by  shaping  the  choices  of  key  states,  preventing 
adversaries  from  acquiring  or  using  Weapons  of  Mass  Destruction,  strengthening  and 
expanding  alliances  and  partnerships,  securing  U.S.  strategic  access  and  retaining 
freedom  of  action,  and  integrating  and  unifying  our  efforts. 

e.  Risks 

Since  it  is  not  possible  to  fund  every  project,  ultimately  choices  must  be 
made.  Here  risk  is  defined  in  terms  of  the  potential  for  damage  to  national  security 
combined  with  the  probability  of  occurrence  and  a  measurement  of  the  consequences 
should  the  underlying  risk  remain  unaddressed.  The  first  risk  is  that  partner  contributions 
to  future  coalition  operations  will  vary  in  size,  composition,  competence,  and  capability. 
Second,  the  risk  strategy  must  account  for  four  dimensions  of  risk: 

Operational  Risks  -  Those  associated  with  the  current  force  executing  the 
strategy  successfully  within  acceptable  human,  material,  financial,  and  strategic  costs. 

Future  Challenges  Risks  -  Those  associated  with  the  Department’s 
capacity  to  execute  future  missions  successfully  against  an  array  of  prospective  future 
challengers. 
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Force  Management  Risks  -  Those  associated  with  managing  military 
forces  fulfilling  the  objectives  described  in  the  NDS.  The  primary  concern  here  is 
recruiting,  retaining,  training,  and  equipping  a  force  and  sustaining  its  readiness. 

Institutional  Risk  -  Those  associated  with  the  capacity  of  new  command, 
management,  and  business  practices. 

3.  Gartner  Group’s  COTS  PA  Product  Landscape 

Applications  for  project,  portfolio  and  resource  management  boost  team 
performance  and  enable  IT  management  and  others  to  access  real-time  data  via 
dashboards  for  prioritization  and  quick  decision  making.  As  clients  look  to  better  manage 
IT  portfolios,  vendors  are  marketing  increasingly  integrated  Project  Portfolio 
Management  (PPM)  suites  for  strategically  aligning  and  planning  IT  projects,  and  for 
controlling  application  portfolio  costs.  The  dynamic  PPM  market  is  poised  for  an  annual 
growth  rate  of  12%  to  20%  through  year  end  2012,  as  users  seek  to  tie  strategies  more 
effectively  to  planning  and  control — including  IT  strategy.  As  shown  in  Figure  16,  the 
top  four  PPM  vendors  worldwide  in  2007,  in  terms  of  total  software  revenue,  were  (in 
order)  Microsoft,  Primavera  Systems,  CA  and  Planview.  These  top  four  vendors  held 
46%  of  the  total  worldwide  PPM  market  share  in  2007,  with  revenue  of  more  than  $512 
million  (copyright  Gartner  Group,  2008). 
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Figure  16.  Top  Four  PPM  Vendors  Worldwide  in  2007 
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Our  research  team  will  research  the  top  four  PPM  solutions  and  a  employ  a 
vertically  focused  Risk  Management  solution  to  evaluate  which  is  the  best  or  if  none  are 
capable  of  providing  an  all-in-one  solution.  Our  analysis  will  not  only  require  program 
management  but  critical  portfolio,  decision,  analysis  and  the  ability  to  provide  DoD 
analysis  capability. 


a.  COTS  PPM  Suite#l :  Microsoft  Project  Portfolio  Management 

Program 


Microsoft  provides  an  interesting  approach  to  PPM,  with  use  of  its  Office 
Project  Portfolio  Server  (OPPS)  2007,  which  assists  organizations  in  realizing  its 
potential  by  identifying,  selecting,  managing,  and  delivering  portfolios  that  align  with 
strategic  priorities. 
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Figure  17.  OPPS  2007  Workflow  Design 


Figure  17  above  includes  an  intuitive  workflow  design  as  part  of  a 
tracking  tool  that  is  designed  for  project/program  managers  to  define  control  points  to 
multiple  workflows.  The  control  points  govern  life  cycle  phases  used  as  a  common 
denominator  to  aggregate  and  report  on  projects  across  various  fields.  The  beauty  of  this 
deliverable  is  that  it  receives  managerial  signature  before  moving  forward  in  the  life 
cycle  phases.  This  audit  functionality  makes  stakeholders  aware  and  accountable  as 
projects  move  from  ideas  to  consideration  to  implementation. 
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Figure  18  below  describes  analysis  techniques  that  create  rapid 
assessments  of  the  impact  on  the  portfolio’s  business  value  and  effectively  communicates 
the  tradeoffs,  including  the  listed  projects  within  a  portfolio. 
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Figure  18.  Analysis  Techniques  for  Rapid  Assessment  of  Impact 
on  the  Portfolio’s  Business  Value 


Figure  19  below  is  an  example  of  a  tracking  tool  from  OPPS,  which  is 
designed  to  measure  and  track  projects,  programs,  and  applications  throughout  their  life 
cycle.  This  provides  the  visibility  to  proactively  identify  potential  issues,  promote 
decision  making,  and  help  ensure  that  the  organization  portfolios  deliver  maximum 
business  value. 
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Figure  19.  Sample  OPPS  Tool  for  Measuring  and  Tracking  Projects/Programs/ Applications 

Throughout  Their  Life  Cycle 

OPPS  is  designed  to  support  open  architecture,  enabling  integration  with 
Microsoft  Office  Project  Server  2007  and  other  PPM  COTS  applications  to  provide 
organizations  with  an  end-to-end  project  portfolio  management  solution.  The  bi¬ 
directional  gateway  enables  administrators  to  link  multiple  Microsoft  Office  Project 
Servers  to  OPPS  2007,  providing  managers  with  a  consolidated  view  of  all  projects 
within  the  organization. 

In  summary,  Microsoft  optimizes  the  use  of  OPPS  and  selects  portfolios 
that  best  align  with  an  organization’s  strategic  priorities.  The  interface  provided  is  user- 
friendly,  and  embeds  best  practice  methodologies  and  analytical  techniques  to  ensure 
selection  of  the  right  investments  for  business  solutions.  Microsoft  shares  its  Corporate 
Project  Solutions,  allowing  organizations  to: 

•  Employ  sophisticated  optimization  algorithms  to  select  the  optimal 
portfolio  under  varying  budget  constraints,  such  as  costs  and  Full  Time 
Equivalents  (FTEs). 
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•  Identify  and  break  the  constraints  prohibiting  the  portfolio  from  reaching 

the  Efficient  Frontier. 

•  Automatically  calculate  the  portfolio’s  alignment  with  an  organization’s 

business  strategy. 

•  Enforce  a  rational,  rather  than  emotional,  approach  to  selecting  portfolios. 

(Microsoft  PPM,  2009)  provides  a  link  to  the  Microsoft  PPM  Web  site  on 

the  Internet. 

b.  COTS  PPM  Suite  #2:  Primavera  Systems  Project  Portfolio 

Management 

Customers  use  Primavera  as  a  project,  resource  and  portfolio  management 
solution  to  propose,  prioritize,  and  select  strategic  investments  and  then  to  plan,  manage 
and  control  the  entire  portfolio  of  projects  through  to  successful  completion.  Primavera 
solutions  are  industry-specific  and  role-based,  with  a  design  powered  to  support  global 
enterprises.  Primavera  PPM  product  can  be  used  to  manage  any  type  of  portfolio, 
including: 

•  IT  project  requests 

•  Applications  for  rationalization 

•  New  product  development 

•  Capital  programs 

As  depicted  in  Figure  20  and  Figure  21,  utilizing  Microsoft  Internet 
Explorer,  Primavera’ s  design  supports  everything  from  data  entry  forms  for  details  about 
a  single  investment,  to  scorecards  for  evaluating  a  set  of  investments  in  a  portfolio,  to 
investor  maps  for  reviewing  and  analyzing  a  portfolio  of  investments,  or  even  evaluating 
a  portfolio  of  portfolios. 
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Figure  20.  Sample  Primavera  Project  Portfolio  Management  Tool 


The  key  aspect  of  Primavera  PPM  is  that  it  is  specifically  designed  for 
proposing,  planning,  and  controlling  portfolios  of  investments  in  a  collaborative  way, 
following  an  objective  and  transparent  process.  These  results  propose  enormous  gains  in 
efficiencies  and  improvements  in  business  performance. 
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In  summary,  Primavera  PPM  provides  portfolio  management  software 
solutions,  with  similar  Microsoft  flexibility  and  infrastructure  for  enterprise  portfolio 
management.  The  unique  functionality,  real-world  enterprise  scalability,  and  unlimited 
configurability  make  Primavera  PPM  a  viable  solution  for  managing  just  about  any  type 
of  portfolio  from  ideas  through  execution.  It  also  is  designed  with  powerful  security 
features  for  clean  user  interface  and  collaboration  among  all  organizational  stakeholders. 
(Primavera  PPM,  2009)  provides  a  link  to  the  Primavera  PPM  Web  site  on  the  Internet. 

c.  COTS  PPM  Suite  #3:  Planview  Portfolio  Management  Program 

Planview  has  designed  tools  to  identify  project  requirements,  manage  the 
scope  of  work,  and  minimize  change.  These  tools  provide  more  visibility  into  how  a 
project  can  stay  on  track  and  deliver  real  business  value.  The  Planview  PPM  COTS 
application  was  designed  to  provide  capabilities  to  ensure: 

•  Optimization  of  resources  enterprise- wide 

•  Integrated  decision  making 

•  Ability  to  view  project  performance  and  perform  trend  analysis 

•  Mitigation  of  risks  and  management  of  changes 

As  depicted  in  Figure  22  below,  Planview  PPM  COTS  application  design 
assists  in  balancing  portfolios,  supporting  management  decisions  for  the  proper  allocation 
of  work  to  the  most  appropriate  personnel.  By  managing  work  with  this  portfolio,  one 
can  deliver  the  highest  business  value  to  projects  across  an  organization. 
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Figure  22.  Planview  PPM  Application  Development  Performance  Portfolio 


In  addition,  project  budgets  must  be  forecasted,  baselined,  and  managed 
throughout  the  project  life  cycle.  With  Planview  Enterprise,  IT  financial  managers,  the 
PMO  and  project  managers  can  collaborate  to  better  forecast  costs  and  monitor  spending 
through  actual  resource  assignments  and  reported  time.  The  key  is  the  establishment  of 
integration  with  existing  financial  and  accounting  systems  with  Microsoft,  as  depicted  in 
Figure  23  below. 
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Configurable  screens  conveniently  display  relevant  data  to  improve  the  efficiency  and 
performance  of  product  and  portfolio  managers. 
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Figure  23.  Planview  Enterprise  Tool  for  Forecasting  Costs  and 

Monitoring  Spending 


Plainview  is  designed  to  help  technology  organizations  deliver  value 
through  a  comprehensive,  systematic  approach  to: 

•  Measuring  and  analyzing  current  performance, 

•  Comparing  actual  results  with  clearly  defined  business 
requirements, 

•  Analyzing  and  documenting  decisions,  and 

•  Efficiently  executing  change  initiatives. 

(Plainview  PPM,  2008)  provides  a  link  to  the  Planview  PPM  Web  site  on 

the  Internet. 
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d.  COTS  PPM  Suite#4:  CA  Project  Portfolio  Management 

The  CA  PPM  approach  facilitates  collaborative  projects  between 
organizations  and  CA,  based  on  repeatable  best  practices  guided  by  forecasted 
implementation  from  leadership. 

During  the  design  and  development  process,  CA  and  CA  strategic  partners 
work  together  with  organizations  to: 

•  Assess  and  prioritize  requirements, 

•  Create  a  requirements  specification  and  project  plan, 

•  Architect  a  process  and  technical  design  and  implementation  blueprint 
to  improve  maturity, 

•  Install,  configure,  integrate  and  test  a  solution  “as  built”  for 
environments  and  improve  processes,  and 

•  Deliver  quality  results  quickly  in  30  days  or  less  in  order  to  achieve 
rapid  time-to-value  and  improved  business  alignment. 

In  addition,  like  any  project,  CA  PPM  focuses  on  strong  sponsorship  from 
senior  management,  along  with  representation  from  the  various  stakeholders  within  the 
organization  who  understand  the  drive  needed  to  help  cultivate  the  business  benefits  of 
enhanced  change.  CA  uses  a  deployment  methodology  which  is  certified  to  align 
organization  PPM  capabilities  and  designs  for  strategic  service  life  cycle  management 
goals.  Below  are  some  CA  focal  points  that  are  designed  for  PPM  solutions: 

a.  Achieve  Rapid  Time-to-Value  through  repeatable  scoping  models 
and  gap  analysis  that  identify  role-based  business  objectives,  requirements  and 
configuration  designs,  plus  deployment  methodologies  that  apply  and  adapt  the  pre¬ 
defined  configurations  of  CA  Services  Accelerators  to  deliver  a  custom  CA  PPM 
solution. 

b.  Implement  Multi-phased  Deployments  with  a  goal  of  60-90  day 
increments  so  you  can  continually  deliver  IT  service  value  and  improve  the  maturity  of 
your  CA  PPM  solution  without  overwhelming  your  organization  with  change  and 
excessive  adoption  challenges. 
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c.  Stabilize  your  Solution  at  the  End  of  Deployment  phases  to 
maximize  your  current  investment.  Consultants  tune  your  CA  PPM  solution,  provide 
administrative  knowledge  transfer  to  your  IT  staff  and  solution  overviews  for  your  end 
users,  assist  with  production  rollout  and  make  recommendations  that  can  speed  adoption 
or  further  develop  your  CA  PPM  solution  maturity  and  business  value. 

d.  With  On-boarding  Services  from  CA  Education,  you  can  achieve 
expected  productivity  gains  and  increase  adoption  rates.  A  robust  selection  of  prepared 
courses  delivered  in  a  variety  of  formats  and  options,  custom-designed  courses,  and  the 
CA  Productivity  Accelerator  (a  context-sensitive,  customizable  training  solution)  provide 
focused,  ongoing  education  support  to  speed  adoption  of  your  CA  PPM  solution. 

The  touted  benefits  of  CA  PPM  are  designed  to  provide: 

•  Lower  risk  from  experience  and  best  practices. 

•  Faster  results  through  incremental  deployment  methodologies,  self- 
service  support  knowledge  and  in-context  and  online  training. 

•  Improved  outcomes  with  thought  leadership  and  proven  experience. 

•  Knowledgeable  administrators  and  users  from  deployment  on- 
boarding,  continuing  education  and  support  guidance. 

(CA  PPM,  2008)  provides  a  link  to  the  CA  PPM  Web  site  on  the  Internet. 

4.  Forrester  Research  COTS  PA  Product  Landscape 

The  Forrester  Wave™  Project  Portfolio  Management  Tools,  Q4  2007  report 
established  CA  as  the  PPM  leader  in  2007,  according  to  an  article  published  18 
December  2007,  by  Mr.  Lewis  Cardin  et  al.  (p.  8).  Forrester  evaluated  fourteen  leading 
PPM  vendors  across  ninety-five  criteria  and  found  that  CA  and  Planview  established 
PPM  leadership  within  the  field  thanks  to  their  wide  choice  of  mature  features  and 
functions.  Forrester’s  COTS  PA  product  research  uncovered  a  market  environment  in 
which: 

•  CA,  Planview,  HP,  Primavera,  and  IBM  lead  the  pack 

•  Compuware,  Oracle,  Serena,  and  Microsoft  offer  competitive  options 

•  SAP  and  Daptiv  lack  the  expected  full  suite  of  out-of-the-box  offerings 
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This  evaluation  of  the  COTS  PPM  Suite  market  is  intended  to  be  a  starting  point 
only.  Readers  are  encouraged  to  view  detailed  product  evaluations  and  adapt  the  criteria 
weightings  to  suit  their  individual  needs  through  the  Forrester™  Wave  Excel-based 
vendor  comparison  tool  (see  Figure  24). 


Risky  Strong 

Bets  contenders  Performers  Leaders 


Figure  24.  Forrester™  Wave  Portfolio  Management  Tools,  Q4’  07 
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As  the  front  runner,  CA  is  not  alone  and,  by  far,  has  not  cornered  the  market  in 
the  PPM  arena.  However,  as  shown  above,  growth  is  taken  seriously  by  all  players  to 
become  the  best  in  an  economy  that  could  use  leadership.  Below  are  company  profiles 
that  contributed  to  their  PPM  leadership  roles: 

•  Microsoft  moves  into  the  leader  zone.  Microsoft  has  had  some  mountains  to 
climb  with  its  integration  of  UMT  (a  premier  portfolio  management 
application)  and  getting  off  its  thick  client  workstation  solution  for  project 
management.  Microsoft  has  succeeded  in  gluing  these  projects  and  portfolio 
management  solutions  together,  along  with  the  advent  of  its  server 
technology.  While  it’s  reporting  capabilities  continue  to  excel  (no  pun 
intended),  its  customers  will  benefit  from  further  development  in  its 
methodology  offering  and  increased  financial  management  capability, 
particularly  in  the  area  of  chargeback  (Microsoft  Advances  As  A  Leader  In 
The  PPM  Market  (Cardin,  Lewis,  Forrester  Wave,  2007). 

•  Primavera  continues  to  keep  a  stronghold  with  its  demand  management, 
workflow,  and  methodology  software;  it  must  develop  more  depth  in  its 
Application  Portfolio  Management  and  infrastructure  portfolio  management 
offerings  to  be  as  equally  attractive  to  its  IT  customers  as  it  is  with  rest  of  the 
enterprise  (Primavera  Is  A  Leader  In  The  Large  Enterprise  PPM  Market, 
(Cardin,  L.,  Cullen,  A.,  &  Cecere,  M.,  2007). 

•  Planview  continues  to  be  in  front  in  the  specific  area  of  portfolio  management 
and  is  neck  and  neck  with  Primavera  on  project  and  portfolio  management 
methodology.  Planview  still  has  some  work  to  do  with  its  integrated  IT 
management  offering  (Planview  Is  An  Undeniable  Leader  In  The  PPM 
Market,  (Cardin,  L.  Cullen,  A.,  &  Cecere,  M.,  2007). 

•  CA’s  continued  strength  in  reporting  and  its  focus  on  enterprise  IT 
management  makes  PPM  an  integral  part  of  overall  customer  IT  management 
(CA  Leads  in  Many  Offerings  in  the  PPM  Market  (Cardin,  L.,  Cullen,  A.,  & 
Cecere,  M.,  2007). 
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5. 


Real  Options  Valuation’s  Risk  Simulator 


Real  Options  Valuation’s  Risk  Simulator  software  package  will  help  to  identify, 
quantify,  and  evaluate  risk  in  projects  and  decisions.  Risk  Simulator,  a  powerful  Excel 
add-in  application,  is  used  for  applying  simulation,  forecasting,  statistical  analysis,  and 
optimization  in  existing  Excel  spreadsheet  models.  It  contains  four  different  modules: 
Monte  Carlo  Simulation,  Optimization,  Statistical  and  Analytical  tools,  Time  Series  and 
Cross-Sectional  forecasting.  Risk  simulator  is  also  integrated  with  the  Real  Options 
Super  Lattice  Solver  software,  for  solving  strategic  real  options,  financial  options,  and 
employee  stock  options. 


Figures  25  through  29  depict  sample  optimization  models  which  were  run  with 
Risk  Simulator,  both  before  and  after  optimization.  Figure  25  is  a  discrete  model  that 
uses  a  static  optimization  which  is  run  on  static  models,  no  assumption  inputs,  without 
simulations.  This  type  of  optimization  is  usually  run  to  determine  the  initial  optimal 
portfolio  before  more  advanced  optimizations  are  applied. 
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ENPV  is  the  expected  NPV  of  each  credit  line  or  project,  while  Cost  can  be  the  total  cost  of 
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administration  as  well  as  required  capital  holdings  to  cover  the  credit  line,  and  Risk  is  the 

21 

Coefficient  of  Variation  of  the  credit  line's  ENPV. 

22 

Figure  25.  Sample  Optimization  Model  Run  with  Risk  Simulator  (Discrete  Model) 


Note  that  the  selection  criteria  are  all  “ones.”  The  decision  criteria  for  this  model 
is  to  select  six  of  the  best  projects  based  on  risk  and  cost,  with  a  cost  limit  of  less  than 
$5000.  Figure  26  below  is  the  same  model  after  being  run  through  250  iterations  and 
evaluating  the  risk/cost  factors  to  make  the  best  decision. 
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Figure  26.  Sample  Optimization  Model  Run  with  Risk  Simulator  (After  Run  Through  250 

Iterations) 

Figure  27  below  is  a  sample  of  a  Stochastic  Model,  which  can  be  run  either  by 
dynamic  optimization  or  by  stochastic  optimization.  Dynamic  optimization  is  a  first-run 
simulation.  The  results  of  the  simulation  are  applied  to  the  model,  and  then  an 
optimization  is  applied  to  the  simulated  values. 
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Specifications  of  the  optimization  model: 
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17 
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Maximize  Return  to  Risk  Ratio  (B13) 
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Decision  Variables: 

Allocation  Weights  (D6:D10) 

19 

Restrictions 
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Variables: 

Minimum  and  Maximum  Required  (E6:F10) 
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Portfolio 

Total  Allocation  Weights  100%  (D12 

is  set  to  100%) 

21 

22 

Additional  specifications: 

23 

24 

1 .  One  can  always  maximize  portfolio  total  returns  or 

minimize  the  portfolio  total  risk. 

25 

2.  Incorporate  Monte  Carlo  simulation  in  the  model  by  simulating  the  returns  and  volatility  of  each  asset  class 

26 

and  apply  Simulation-Optimization  techniques. 

27 

3.  The  portfolio  can  be  optimized  a 

s  is  without  simulation  using  Static  Optimization  techniques. 

Figure  27.  Sample  Optimization  Model  Run  with  Risk  Simulator 

(Stochastic  Model) 
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Figure  28  below  is  the  same  model  run  through  the  dynamic  optimization  with 


1000  simulation  trials. 
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Specifications  of  the  optimization  model: 

16 

17 

Objective: 

Maximize  Return  to  Risk  Ratio  (Bit. 

9 

IS 

Decision  Variables: 

Allocation  Weights  (D6D10) 

19 

Restrictions  on  Decision  Vari  Minimum  and  Maximum  Reauired  (E6.F10) 

20 

Constraints: 

Portfolio 

Total  Allocation  Weights  100%  (D12  is  set  to  100%) 

21 

22 

Additional  specifications: 

2  3 

24 

1.  One  can  always  maximize  portfolio  total  returns 

i  or  minimize  the  portfolio  total  risk. 

25 

2.  Incorporate  Monte  Carlo  simulation  in  the  model  by  simulating  the  returns  and  volatility  of  each  asset  class 

26 

and  apply  Simulation-Optimization  techniques. 

27 

3.  The  portfolio  can  be  optimized  as  is  without  simulation  using  Static  Optimization  techniques. 

Figure  28.  Sample  Optimization  Model  Run  with  Risk  Simulator 
(Dynamic  Optimization  with  1000  Simulation  Trials) 


Note  that  the  rankings  have  changes  based  on  the  simulations  comparing  the  input 
assumptions  (green  boxes)  and  the  decision  criteria  (blue  box).  The  light  blue  boxes  are 
the  objective,  risk  return  ratio  (B13),  and  the  constraint  (100%,  (D12)). 

Stochastic  optimization  is  similar  to  dynamic  optimization,  except  that  the  process 
is  repeated  several  times.  The  final  decision  variables  will  each  have  their  own  forecast 
chart  indicating  their  optimal  range.  Figure  29  below  is  the  same  model  run  through  the 
stochastic  optimization  with  1000  simulation  trials  and  20  optimization  runs. 
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19 

Restrictions  on  Decision  Var  Minimum  and  Maximum  Required  (E6:F10) 

20 

Constraints: 

Portfolio  Total  Allocation  Weights  100%  (D12  is  set  to  100%) 

21 

22 

Additional  specifications: 

23 

24 

1.  One  can  always  maximize  portfolio  total  returns  or  minimize  the  portfolio  total  risk. 

25 

2.  Incorporate  Monte  Carlo  simulation  in  the  model  by  simulating  the  returns  and  volatility  of  each  asset  class 

26 

and  apply  Simulation-Optimization  techniques. 

27 

3.  The  portfolio  can  be  optimized  as  is  without  simulation  using  Static  Optimization  techniques. 

Figure  29.  Sample  Optimization  Model  Run  with  Risk  Simulator  (Stochastic  Model  with 

1000  Simulation  Trials  and  20  Optimization  Runs) 


Again,  note  the  changes  in  the  rankings  and  the  objective. 


Figure  30  below  depicts  the  statistical  results  of  the  optimization  for  Program  3, 
with  20  data  points. 


S3  Program  3 

84  NumtjerofDafapOiiifs 

85  Wean 

86  Median 

87  Standard  Deviation 

88  Variance 

89  4'/erage  Deviation 

90  Maximum 

91  Minimum 

92  Range 

93  Skewness 

94  Kurtosis 

95  25%  Perrenffie 

96  75%  Pencenffj'e 

97  Ernor  Precision  at  95% 

_ _ 


Figure  30.  Statistical  Results  of  Optimization  for  Program  3  (with  20  Data  Points) 
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The  Risk  Simulation  software  provides  a  risk/decision  analysis  to  help  any 
business  make  better  decisions,  both  now  and  in  the  future. 

Efficient  Frontier  (EF)  analysis,  shown  in  Figure  31  below,  is  the  process  of  re¬ 
running  multiple  optimizations  with  different  constraints  and  each  optimal  portfolio  is  a 
point  on  the  frontier.  To  run  the  analysis,  click  on  the  Constraints  icon  or  in  Risk 
Simulator,  select  Optimization  and  Constraints.  Then  add  as  many  changing  constraints 
as  desired  to  generate  multiple  EFs. 


Figure  3 1 .  Efficient  Frontier 


6.  Palisade  @RISK  (http://www.palisade.com/) 

Palisade  @RISK  performs  risk  analysis  using  Monte  Carlo  simulation  to  show 
you  many  possible  outcomes  using  Microsoft  Excel  spreadsheets,  and  tells  you  how 
likely  outcomes  are  to  occur.  You  can  then  decide  which  risks  are  worth  taking  and 
which  ones  to  avoid,  allowing  for  improved  decision  making  under  uncertainty.  @RISK 
uses  simulation  to  answer  questions  like: 

a.  “What  is  the  probability  of  profit  exceeding  $  1 0  million?” 

b.  “What  are  the  chances  of  losing  money  on  this  venture?” 
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c.  “What  is  the  probability  that  the  project  will  be  delivered  within  budget?” 

d.  “How  much  contingency  (management  reserve)  should  be  included?” 

@RISK  is  an  add-in  to  Microsoft  Excel,  integrating  completely  with  the 
spreadsheet.  All  @RISK  functions  are  Excel  functions,  and  behave  exactly  the  same  as 
native  Excel  functions.  @RISK  windows  are  all  linked  directly  to  cells  in  your 
spreadsheet,  so  changes  in  one  place  are  carried  over  to  the  other.  @RISK  graphs  point  to 
their  cells  via  callout  windows.  Figure  32  depicts  an  @RISK  probability  distribution  for  a 
sample  project. 


Figure  32.  @RISK  Probability  Distribution 


7.  Shortcomings  of  COTS  Products 

The  drive  to  incorporate  COTS  software  and  hardware  is  often  based  on 
incomplete  or  inaccurate  information.  Clearly  identified  requirements  that  cannot 
incorporate  COTS  software  and  hardware  need  additional  research,  cost,  and 
development  within  their  product  design.  However,  using  PPM  COTS  products  does 
offer  many  advantages  and  disadvantages.  For  instance,  many  of  these  shortcomings  are 
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described  by  engineers  who  are  reluctant  to  move  to  COTS,  if  for  no  other  reason  than  a 
general  resistance  to  change.  Many  engineers  display  the  attitude  “if  we  did  not  make  it; 
the  product  design  is  no  good.”  It  is  basically  that  type  of  mentality  that  promotes  a 
general  distrust  of  COTS  products. 

a.  Advantages  of  COTS 

The  advantages  associated  with  the  use  of  COTS  are: 

•  Immediately  available  and  shorter  development  schedule 

•  Reduced  cost 

•  Increased  portability 

•  Improved  Quality  (resulting  from  more  efficient  testing) 

b.  Disadvantages  of  COTS 

The  disadvantages  associated  with  the  use  of  COTS  are: 

•  Hard  to  meet  special  requirements 

•  Continual  investment  in  COTS  product 

•  Bad  interoperability 

Although  it  is  generally  a  good  idea  not  to  accept  a  product  without  verifying  its 
capabilities,  it  is  also  not  a  good  idea  to  reject  a  product  on  general  principle.  Below  are 
the  shortcomings  of  discussed  COT  products  and  assumptions  about  their  acceptable 
levels  of  requirements  that  need  to  be  or  have  been  adjusted  for  product/solution. 

a.  Microsoft’s  design  lacked  Portfolio  management  requirements  to 
gather  accurate  data,  which  led  to  the  acquisition  of  UMT.  This  compensated  for  the 
needed  solution  and  supported  the  project  management  software  deployed  in  late  2007. 

b.  Primavera  has  mastered  the  art  of  determining  trends  in  the  PPM 
market,  identifying  the  PPM  needs  of  its  target  markets,  and  acquiring  solutions  that  fit 
the  vendor’s  ever  changing  and  evolving  products.  The  design  has  shortcomings  that  are 
ideally  suited  to  large  enterprises  with  mature  PPM  processes;  design  is  not  suited  for 
requirements  of  smaller-sized  businesses. 
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c.  Plainview’s  design  is  the  second  strongest  portfolio  management 
solution  respected  on  the  market. 

d.  CA  is  the  leader  and  continuously  moving  up  in  its  leadership  position. 
Shortcomings  are  very  limited  to  its  continued  investments  in  integrated  IT  management. 

C.  MODEL  DEVELOPMENT 


1.  Model  Data 


Using  the  basic  data  depicted  in  Figure  33  below,  our  research  team  attempted  to 
develop  a  model  in  (1)  Real  Options  Valuation’s  Risk  Simulator  and  (2)  @RISK  that 
would  analyze  the  data  and  provide  usable  output  as  measures  of  EMV.  We  also 
attempted  to  model  this  data  with  the  Program  Management  (PM)  software  but  were 
unable  to  do  so.  The  EMV  measures  for  this  model  were  developed  by  our  team. 
However,  in  practice,  the  EMV  measures  should  be  developed  by  Budget,  Strategic,  and 
Acquisition  professionals  to  ensure  their  accuracy. 


Program 

ENPV 

NPV 

Cost 

Strategy  ” 
Ranking 

P-1  i  1  ita  ry 

Score 

Tactical 

Score 

FTE 

Resoucres 

Program  1 

$501.60 

$102.30 

$443.50 

1.94 

2.51 

1.77 

3.91 

Program  2 

$726.30 

$121.30 

$353.30 

3.75 

3.84 

3.56 

2.73 

Program  3 

$932.40 

$354.20 

$925.90 

2.34 

3.12 

3.24 

5.24 

Program  4 

$620.50 

$265.30 

$512.10 

1 .36 

5.14 

5.1 

3.41 

Program  5 

$420.80 

$271.20 

$524.50 

1.32 

3.22 

1.62 

3.29 

Program  6 

$2,087.50 

$345.90 

$1,368.70 

4.63 

6.37 

6.21 

4.56 

Program  7 

$3,264.90 

$594.10 

$1,602.10 

10.59 

3.69 

7.53 

4.23 

Program  3 

$2,563.40 

$421.50 

$2,133.40 

2.33 

2.64 

4.12 

5.61 

Program  9 

$5,463.10 

$1,032.20 

$2,712.30 

7.63 

6.94 

7.99 

4.68 

Program  10 

$3,243.90 

$775.40 

$2,502.30 

5.74 

5.53 

6.23 

6.43 

Program  11 

$125.90 

$32.30 

$28.60 

15.26 

9.34 

9.17 

5.47 

Program  12 

$235.10 

$85.60 

$109.50 

6.42 

4.25 

3.65 

3.91 

Program  13 

$4,163.50 

$325.10 

$3,054.20 

5.26 

3.41 

6.21 

5.23 

Program  14 

$4,463.70 

$594.10 

$1,673.80 

10.54 

8.46 

7.54 

4.23 

Program  15 

$587.60 

$421.50 

$409.00 

2.81 

2.74 

3.52 

5.34 

Program  16 

$1,698.40 

$431.90 

$653.30 

11.92 

3.45 

3.37 

4.26 

Program  17 

$758.40 

$134.20 

$435.50 

4.65 

4.52 

4.35 

6.32 

Program  18 

$1,243.30 

$301.50 

$331.50 

4.35 

6.13 

5.62 

3.57 

Program  19 

$389.90 

$26.10 

$159.00 

7.93 

7.65 

3.62 

4.93 

Program  20 

$3,364.70 

$1,257.60 

$1,453.70 

15.46 

9.43 

9.34 

12.64 

$21,952.20 

127.33 

112.93 

114.26 

100.04 

Maximize 

<=$10,500 

<=  100 

<=  30 

Objective:  Maximize  Total  Portfolio  Rot  a  ms  times  the  Portfolio  Comprehensive  Score 

Decision  Variables:  Allocation  or  Go/No-Go  Decision 

Restrictions  on  Decision  Variables  Binary  decision  variables  ( 0  or  1) 

Constraints:  Total  Cost  is  less  than  $10,500  (in  thousands  or  millions  of  dollars) 

Less  than  or  equal  to  10  projects  selected  in  the  entire  portfolio 

Full-time  Equivalence  resources  have  to  be  less  than  80 

Total  Strategic  Ranking  for  the  entire  portfolio  must  be  less  than  100 

Figure  33.  Base  Evaluation  Model 
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Earned  Net  Present  Value  is  an  enhancement  of  the  NPV  that  explicitly  addresses 
uncertainty.  NPV  compares  a  single  stream  of  cash  flows  in  today’s  dollars  to  the  value 
of  that  same  dollar  in  the  future.  Cost  is  the  actual  cost  listed  in  the  2009  Defense 
Procurement  Budget  Request.  Strategic  Ranking,  Military  Score,  and  Tactical  Score  are 
EMV  measures  developed  by  evaluating  the  NDS  and  then  scoring  the  programs  based 
on  how  they  meet  the  NDS.  FTE  resources  equates  to  the  amount  of  actual  resources 
used  as  a  percent.  One  hundred  percent  means  that  all  resources  are  fully  utilized  all  the 
time.  The  goal  is  to  maximize  the  portfolio  returns  without  exceeding  an  arbitrary  budget 
of  $10,500  while  keeping  the  strategic  ranking  below  100  and  the  FTE  below  80. 

In  evaluating  this  model,  we  plan  to  verify  that  PA  and  Defense  budget  decision 
making  can  be  improved  using  COTS  software. 

2.  Real  Options  Valuation’s  Risk  Simulator 

Figure  34  is  the  model  developed  using  the  Real  Options  Risk  Simulator.  The 
task  was  to  run  an  optimization/simulation  to  determine  which  ten  of  the  following 
twenty  programs  best  meet  the  requirements  outlined  in  the  NDS.  Only  ten  programs 
will  go  forward.  These  are  real  programs  in  the  FY  2009  budget  with  the  real  costs 
included.  The  names  of  the  real  programs  will  be  revealed  later  in  the  analysis  section. 
The  ranking  and  military/tactical  scores,  along  with  the  FTE  resources,  are  based  on  the 
NDS  and  the  President’s  goals  are  located:  http://www.whitehouse.gov/agenda/defense/. 

Because  our  thesis  is  based  on  PA,  we  will  attempt  to  run  this  through  PPM 
programs  as  well  as  risk  simulators.  For  the  purpose  of  this  model,  the  military  score, 
comprehensive  score,  and  tactical  score  are  all  measures  of  EMV. 

After  running  the  discrete  (static)  optimization  on  this  model  with  the  original 
budget  of  less  than  $10,500,  and  a  strategy  rank  of  less  than  100,  no  more  than  ten 
programs,  and  FTE  resources  not  to  exceed  80. 
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Military  Portfolio  Optimization 

Project  Name 

ENPV 

NPV 

Cost 

Strategy 

Ranking 

Return  to 
Rank  Ratio 

Profitability 

Index 

Selection 

Military  Score 

Tactical 

Score 

FTE 

Resources 

Comprehensive 

Score 

Project  1 

$501.60 

$102.30 

$443.50 

4.26 

117.75 

1.23 

1 

2.51 

1.77 

3.91 

1.40 

Project  2 

$726.30 

$121.30 

$358.80 

5.61 

129.47 

1.34 

1 

3.84 

3.56 

2.73 

1.77 

Project  3 

$932.40 

$354.20 

$ 92590 

2.34 

398.46 

1.38 

1 

3.12 

3.24 

5.24 

2.12 

Project  4 

$ 62050 

$ 265  80 

$ 512.10 

3.21 

193.30 

1.52 

1 

5.14 

5.10 

3.41 

2.46 

Project  5 

$420  80 

$ 271.20 

$ 524.50 

1.32 

318  79 

1.52 

1 

3.22 

1.62 

3.29 

1.60 

Project  6 

$2,087.50 

$ 345  90 

$1,868.70 

4.68 

446.05 

1.19 

1 

6.37 

6.21 

4.56 

2.98 

Project  7 

$3,264.90 

$594  10 

$1,602.10 

10.59 

308.30 

1.37 

1 

8  69 

7.53 

4.23 

3.43 

Project8 

$2,563.40 

$42150 

$2,133.40 

2.83 

90580 

1.20 

1 

2.64 

4.12 

5.61 

2.31 

Project  9 

$5,468.10 

$1,032.20 

$2,712.30 

7  68 

71199 

1.38 

1 

6.94 

7.99 

4.68 

3.46 

Project  10 

$3,248.90 

$775.40 

$2,502.80 

5.74 

566.01 

1.31 

1 

5.53 

6.23 

6.43 

3.16 

Project  1 1 

$ 125.90 

$32  80 

$28.60 

15.26 

8.25 

2.15 

1 

9.84 

9.17 

5.47 

4.13 

Project  12 

$ 235.10 

$85  60 

$ 10950 

6.42 

36.62 

1.78 

1 

4.25 

3.65 

3.91 

2.00 

Project  13 

$4,163.50 

$325.10 

$3,054.20 

5.26 

791.54 

1.11 

1 

3.41 

6.21 

5.23 

2.77 

Project  14 

$4,468.70 

$594  10 

$1,673.80 

1054 

8.46 

7.54 

1 

8.46 

7.54 

4.23 

3.41 

Project  15 

$587  60 

$42150 

$ 40900 

2.81 

209.11 

2.03 

1 

2.74 

3.52 

5.34 

2.13 

Project  16 

$1,698.40 

$431.90 

$658.30 

11.92 

142.48 

1.66 

1 

8.45 

8.87 

4.26 

3.74 

Project  17 

$ 75840 

$134.20 

$ 48550 

4.65 

163.10 

1.28 

1 

4.52 

4.35 

6.32 

2.60 

Project  18 

$1,248.30 

$301  50 

$331.50 

4  85 

257.38 

1.91 

1 

6.13 

5.62 

3.57 

2  66 

Project  19 

$ 389.90 

$26.10 

$159.00 

7.98 

48.86 

1.16 

1 

7.65 

8.62 

4.98 

3.73 

Project  20 

$3,864.70 

$1,257.60 

$1,458.70 

15.46 

249.98 

1.86 

1 

9.48 

9.34 

12.64 

5.21 

Total 

$37,374.90 

$21,952.20 

133.41 

1  20  1 

112.93 

114.26 

100.04 

57.08 

Profit/Rank 

Profit*Score 

$280.15 

$2,133,370.83 

Maximize 

< =$10500 

<=100 

x  <=10 

<=80 

Figure  34.  Real  Programs  in  the  FY09  Budget  with  Costs 

Figure  35  shows  the  model  after  the  initial  optimization  run  including  the  ten 
selected  programs.  The  model  was  then  run  several  more  times,  changing  the  constraints 
of  budget,  and  number  of  programs  allowed.  This  information  was  then  used  to  update 
the  data  in  the  efficient  frontier  charts  below. 
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Project  Name 

ENPV 

NPV 

Cost 

Strategy 

Ranking 

Return  to 
Rank  Ratio 

Profitability 

index 

Selection 

Military  Score 

Tactical 

Score 

FTE 

Resources 

Comprehensive 

Score 

Project  1 

$501.60 

$102.30 

$443.50 

4.26 

117.75 

1.23 

0 

2.51 

1.77 

3.91 

1.40 

Project  2 

$726.30 

$121.30 

$358.80 

5.61 

129.47 

1.34 

0 

3.84 

3.56 

2.73 

1.77 

Project3 

$932.40 

$354.20 

$925.90 

2.34 

396.46 

1.38 

0 

3.12 

3.24 

5.24 

2.12 

Project  4 

$620.50 

$265.80 

$512.10 

3.21 

193.30 

1.52 

0 

514 

5.10 

3.41 

2.46 

Project  5 

$420.80 

$271.20 

$524.50 

1.32 

318.79 

1.52 

0 

3.22 

1.62 

3.29 

1.60 

Project  6 

$2,087.50 

$345.90 

$1,868.70 

4.68 

446.05 

1.19 

0 

6.37 

6.21 

4.56 

2.98 

Project  7 

$3,264.90 

$594.10 

$1,602.10 

10.59 

308.30 

1.37 

1 

8.69 

7.53 

4.23 

3.43 

Project 

$2,563.40 

$421.50 

$2,133.40 

2.83 

905.80 

1.20 

0 

2.64 

4.12 

5.61 

2.31 

Project  9 

$5,468.10 

$1,032.20 

$2,712.30 

7.68 

711.99 

1.38 

1 

6.94 

7.99 

4.68 

3.46 

Project  10 

$3,248.90 

$775,40 

$2,502.80 

5.74 

566.01 

1.31 

1 

5.53 

6.23 

6,43 

3.16 

Project  11 

$125.90 

$32.80 

$28.60 

15.26 

8.25 

2.15 

1 

9.84 

9.17 

5.47 

4.13 

Project  12 

$235.10 

$85.60 

$ 109.50 

6.42 

36.62 

1.78 

1 

4.25 

3.65 

3.91 

2.00 

Project  13 

$4,163.50 

$325.10 

$3,054.20 

5.26 

791.54 

1.11 

0 

3.41 

6.21 

5.23 

2.77 

Project  14 

$4,468.70 

$594.10 

$1,673.80 

10.54 

8.46 

7.54 

1 

8.46 

7.54 

4.23 

3.41 

Project  15 

$587.60 

$421.50 

$409.00 

2.81 

209.11 

2.03 

0 

2.74 

3.52 

5.34 

2.13 

Project  16 

$1,698.40 

$431.90 

$658.30 

11.92 

142.48 

1.66 

1 

8.45 

8.87 

4.26 

3.74 

Project  17 

$758.40 

$134.20 

$485.50 

4.65 

163.10 

1.28 

1 

4.52 

4.35 

6.32 

2.60 

Project  16 

$1,248.30 

$301.50 

$331.50 

4.85 

257.38 

1.91 

1 

6.13 

5.62 

3.57 

2.66 

Project  19 

$389.90 

$26.10 

$159.00 

7.98 

48.86 

1.16 

1 

7.65 

8.62 

4.98 

3.73 

Project  20 

$3,864.70 

$1,257.60 

$1,458.70 

15.46 

249.98 

1.86 

0 

9.48 

9.34 

12.64 

5.21 

Total 

$2 0,906.60 

$10,263.40 

85.63 

1  To  | 

70A6 

69.57 

48.08 

32.32 

Profit/Rank 

$244.15 

ProftScore 

$675,768.93 

Maximize 

<=$10500 

<=100 

x<=10 

<=80 

F igure  3  5 .  Optimization  with  Ten  ( 1 0)  Programs 
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Figure  36  is  a  static  efficiency  frontier  showing  the  EMV  and  ROI  for  the 
different  constraint  levels.  This  model  was  modified  to  include  the  percent  ROI  at  the 
different  constraint  levels.  Note  that  the  baseline  is  annotated  in  red  with  an  ROI  of  zero. 


Static  Efficiency  Frontier 


Comprehensive 

Tactical 

Military  Score 

Allowed 

ROI- RANK 

Percent 

Budget 

Score 

Score 

Projects 

Objective 

Change  ROI 

$9.500  00 

32.37 

69.03 

70.16 

9 

$689,082.38 

1.97% 

$10,500.00 

32.32 

69.57 

70.46 

10 

$675,768.93 

0.00% 

$11,500.00 

35.82 

72  59 

73.34 

11 

$810,419.62 

19.93% 

$12,500.00 

39.27 

81.77 

80.68 

12 

$984,108.39 

45.63% 

$1 3,000.00 

40  59 

84.01 

82.20 

13 

$ 998,868  72 

47.81% 

Baseline 


Comprehensive  Score 


42.00 

40.00 

38.00 

36.00 

34.00 

32.00 

30.00 


90.00 

85.00 

80.00 

75.00 

70.00 

65.00 


S8.500  S9,000  59,500  510,000  510,500511,000  511,500512,000  512,500513,000  513,500 
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Figure  36.  Efficient  Frontiers  with  ROI 
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Figure  37.  Optimal  Solution  Budget  with  Twelve  Programs 
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Figure  37  shows  the  optimal  solution  based  on  the  efficient  frontier  and  ROI. 
This  solution  will  be  covered  in  detail  in  the  analysis  section. 

3.  Real  Options  Valuation’s  Risk  Simulator  Data  Analysis 

The  analysis  of  the  risk  simulator  model  shows,  without  a  doubt,  that  there  is  a 
definite  efficient  frontier  in  which  there  is  a  substantial  ROI  limit  in  the  profile  used  in 
this  model.  As  shown  in  Figure  38  below,  from  $9,500  to  $10,500  there  is  actually  a 
decrease  in  the  ROI  maximization  objective.  From  $10,500  to  $12,500  there  is  a 
substantial  increase  in  the  EMV  and  ROI  objectives  with  a  rapid  slow  down  in  ROI 
above  that  threshold.  Note  that  by  increasing  the  budget  by  $2,000  and  allowing  for 
additional  programs,  decision  makers  are  given  the  “opportunity”  to  increase  the  Defense 
capabilities  outlined  in  the  NDS.  This  “opportunity”  is  one  of  the  options  available  to 
Defense  decision  makers.  The  option  does  not  have  to  be  exercised,  but  it  is  available  if 
needs  dictate  and  funding  is  available  from  Congress.  Other  options  include  using  eleven 
programs  or  thirteen  programs,  if  funding  becomes  available,  or  even  using  nine 
programs  in  the  case  of  budget  cuts.  Note  that  using  nine  programs  provides  an  even 
better  ROI  than  the  baseline. 
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Figure  38.  Efficient  Frontier  ROI  from  Baseline 


This  tells  the  decision  makers  in  Washington  that,  based  on  the  NDS,  a  budget  of 
$10,500  is  not  the  most  optimal  to  provide  for  our  Defense  needs.  If  more  funding  were 
allotted,  the  most  optimal  solution,  based  on  strategic  value,  would  be  a  budget  of 
$12,500  and  twelve  programs.  Anything  more  would  be  a  waste  of  taxpayers’  money  by 
funding  a  low  ROI,  and  anything  less  would  decrease  the  value  of  the  Defense  plan  and 
make  the  budget  less  effective,  while  decreasing  Defense  capabilities. 
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Figure  39  shows  the  programs  that  were  evaluated  by  name.  The  programs  in 
green  are  programs  selected  at  both  the  ten  and  twelve  program  optimization  levels,  while 
the  programs  in  orange  were  specifically  from  the  ten  program  optimization,  and  the 
programs  in  blue  were  from  the  twelve  program  optimization. 
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Figure  39.  Real  Programs  by  Name 


Infigure  39,  it  is  interesting  to  note  that  at  the  $10,500  level  the  program  selected 
the  DDG-1000  program  and  the  Marine  High  Mobility  Rocket  System  in  addition  to  the 
other  programs  in  green.  This  is  based  on  the  limited  budget  and  EMVs  utilized.  At  the 
other  end  of  the  spectrum,  we  have  included  more  Patriot  Missiles,  Air  Force  and  Marine 
V-22’s  and  additional  Special  Operations  personnel.  It  is  obvious  that  the  increased  cost 
was  one  of  the  reasons  they  were  not  in  the  initial  evaluation,  but  they  bring  a  higher  ROI 
to  the  Defense  capability  requirements.  Additionally,  it  should  be  noted  that  both  the 
DDG  (Guided  Missile  Destroyer)  program  and  the  Marine  High  Mobility  Rocket  System 
have  been  dropped.  In  examining  the  data,  it  is  obvious  that  the  DDG  program  was  too 
expensive  and  that  the  Marine  High  Mobility  Rocket  System  did  not  provide  a  high 
enough  return  to  be  included  in  the  Defense  budget. 

The  programs  that  made  the  cut  during  both  evaluations,  the  Joint  Strike  Fighter 
and  the  Global  Hawk  Unmanned  Aerial  Vehicle  system,  made  the  cut  based  on  their 
capabilities  rating  and  ability  to  meet  the  future  needs  of  the  Defense  Department. 


58 


Programs  not  selected  in  either  evaluation  were  either  cost  prohibitive  or  did  not  provide 
the  capabilities  required  to  meet  security  requirements.  Note:  The  modeling  data  for 
@Risk  is  provided  in  Appendix  B. 
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III.  RECOMMENDATIONS 


A.  GAO 

The  GAO  recommends  that  the  Secretary  of  Defense  implement  an  enterprise¬ 
wide  portfolio  management  approach  to  making  weapon  system  investments  that 
integrates  the  assessment  and  determination  of  war  fighting  needs  with  available 
resources  and  cuts  across  the  services  by  functional  or  capability  area  (GAO,  2008).  To 
ensure  the  success  of  such  an  approach,  the  Secretary  should  establish  a  single  point  of 
accountability  at  the  Department  level  with  the  authority,  responsibility,  and  tools  to 
ensure  that  portfolio  management  for  weapon  system  investments  is  effectively 
implemented  across  the  Department.  In  addition,  the  Secretary  should  ensure  that  the 
following  commercial  best  practices,  identified  in  this  report,  are  incorporated: 

•  Implement  a  review  process  in  which  needs  and  resources  are  integrated 
early  and  in  which  resources  are  committed  incrementally  based  on  the 
achievement  of  specific  levels  of  knowledge  at  established  decision 
points; 

•  Prioritize  programs  based  on  the  relative  costs,  benefits,  and  risks  of  each 
investment  to  ensure  a  balanced  portfolio; 

•  Require  increasingly  precise  cost,  schedule,  and  performance  information 
for  each  alternative  that  meets  specified  levels  of  confidence  and 
allowable  deviations  at  each  decision  point  leading  up  to  the  start  of 
product  development; 

•  Establish  portfolio  managers  who  are  empowered  to  prioritize  needs,  make 
early  go/no-go  decisions  about  alternative  solutions,  and  allocate  resources 
within  fiscal  constraints;  and 

•  Hold  officials  at  all  levels  accountable  for  achieving  and  maintaining  a 
balanced,  joint  portfolio  of  weapon  system  investments  that  meet  the 
needs  of  the  war  fighter  within  resource  constraints. 

The  GAO  also  recommends  that  the  Secretary  take  steps  to  support  Department- 
level  decision  makers  and  portfolio  managers  by  developing  a  stronger  joint  analytical 
capability  to  assess  and  prioritize  war  fighting  needs. 
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B.  GARTNER  GROUP 


Gartner  Group  recommendations  for  end  users  implementing  PA  solutions 
include  the  following: 

•  Organizations  seeking  improved  PPM  should  identify  organizational  and 
individual  “readiness”  to  adopt  the  discipline  of  portfolio  management, 
through  a  survey  of  capabilities  and  skills  that  are  “foundational”  to  the 
use  and  application  of  portfolio  management,  before  exploring  which  tools 
can  best  support  and  enhance  PPM  capabilities.  In  this  way,  they  can  help 
ensure  that  such  projects  are  successful. 

•  Organizations  should  insist  on  modular,  progressive  implementations  that 
will  fit  their  immediate  requirements,  as  well  as  supporting  them  as  they 
evolve  over  time. 

•  Organizations  should  be  prepared  to  invest  in  ongoing  consulting  support 
to  ensure  that  the  software  is  accepted  by  all  stakeholders  after 
implementation. 

•  Larger  vendors  will  continue  to  be  best-positioned  to  sustain  themselves 
by  cross-selling  from  large  software  portfolios  to  installed  bases.  These 
vendors’  capability  to  reach  down  market  to  small  and  midsize  businesses 
through  established  channel  partners  will  increase.  The  market  will  see 
increased  competition  from  alternative  solutions,  such  as  open  source  and 
Software  as  a  Service,  as  many  vendors  expand  product  offerings  and 
collide  with  other  vendors. 

C.  RESEARCH  TEAM 

The  authors  of  this  paper  offer  the  following  recommendations: 

•  The  PA  toolsets  are  needed  to  make  objective  comparisons  when  working 
with  choices  containing  numerous  or  complex  input  variables  and 
unknown  risk.  As  such,  these  tools  are  essential  for  project  and  program 
managers  who  are  working  with  a  multitude  of  projects  or  programs. 
Recommendation:  NPS  should  explore  the  interest  and  potential  for 
developing  (or  purchasing)  a  program  management  level  curriculum  for 
choosing  and  using  PA  and  PM  tools.  The  point  of  this  curriculum  would 
be  to  educate  and  indoctrinate  future  program  managers  in  how  to  make 
defense  management  decisions  based  on  acceptable  risk  levels  utilizing 
PM  guidance  and  PA  decision  making  tools. 

•  The  choices  of  PA  tools  are  numerous,  and  selection  criteria  can  be  varied 
across  organizations.  As  PA  tools  proliferate  through  DoD  organizations 
there  is  a  potential  for  implementation  of  a  wide  variety  of  (proprietary) 
PA  tool  suites,  which  would  lead  to  costly  support  and  interoperability 
issues.  Recommendation:  NPS  should  explore  the  potential  for 
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development  of  a  PA/PM  Lab  to  be  used  for  product  evaluation,  source 
selection  criteria  development,  and  making  recommendations  to  defense 
officials  on  which  products  to  support. 

•  A  DoD  umbrella  organization  is  needed  to  establish  policies,  guidelines, 
best  practices,  and  provide  PA  implementation  planning  assistance. 
Recommendation:  Building  upon  the  PA  Lab  concept  in  Recommendation 
“b”,  develop  an  NPS/PA  Center  of  Excellence  (CoE)  in  Monterey.  This 
CoE  would  serve  to  coordinate  and  educate  PA  practitioners  and  provide  a 
“Net  Centric”  environment  to  tie  Joint  PA  efforts  together.  The  CoE  is 
part  of  the  solution  to  satisfy  the  GAO  recommendation:  “The  GAO  also 
recommends  that  the  Secretary  take  steps  to  support  department-level 
decision  makers  and  portfolio  managers  by  developing  a  stronger  joint 
analytical  capability  to  assess  and  prioritize  warfighting  needs.” 

In  summary,  the  recommendations  are:  1)  develop  PA  curriculum  and  education 
programs,  2)  develop  a  PA  solutions  lab,  and  3)  develop  a  PA  CoE. 
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IV.  CONCLUSIONS 


A.  COMPARING  THE  EVALUATION  MODEL  WITH  REALITY 


After  developing  the  evaluation  model,  running  simulations,  and  performing  data 
analysis,  the  research  team  has  concluded  that  Risk  Simulator  is  a  very  capable  Microsoft 
Excel  plug-in  which  can  evaluate  program  risk,  evaluate  EMVs,  and  optimize  budgeting 
and  programming  constraints,  all  within  the  scope  of  the  NDS.  This  software  also  has 
hundreds  of  risk  models  built  in,  including  the  military  model.  These  models  are  easily 
modifiable  to  fit  any  number  or  programming  requirements.  To  verify  this  assessment, 
refer  to  Figure  40,  the  original  model  with  the  real  program  names. 
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PIS 

C17  Cargo  Plane 

$1,248.30 

$301.50 

$331.50 

4.85 

6.13 

5.62 

3.57 

P19 

Defense  Space  Recon  Program 

$389.90 

$26.10 

$159.00 

7.98 

7.65 

8.62 

4.98 

P2Q 

Special  Operations  Command 

$3,864.70 

$1,257.60 

$1,458.70 

15.46 

9.48 

9.34 

12.64 

Figure  40.  Evaluation  Model  with  Real  Program  Names 


On  6  April  2009,  Secretary  of  Defense  Gates  held  a  press  conference  outlining  the 
2010  Defense  Budget  request.  To  the  astonishment  of  Congress,  the  Defense  Industrial 
Complex,  and  defense  contractors  it  was  announced  that  he  planned  to  cut  the  F-22 
fighter  program,  stop  building  the  DDG-1000,  increase  production  of  the  Joint  Strike 
Fighter  and  increase  the  number  of  Special  Operations  personnel.  These 
recommendations  are  directly  in  line  with  the  evaluation  results  produced  by  Real 
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Options  Valuation’s  Risk  Simulator.  Although  there  are  some  differences  in  the  actual 
budget  request  and  our  evaluation,  they  would  be  consistent  if  we  had  known  the  EMV 
scoring  used  by  the  Secretary. 

B.  MODELING  USING  COTS  PRODUCTS 

Gartner  and  Forrester  identified  several  PPM  software  suites  which  can  be  used  to 
develop  and  manage  project  models.  However,  these  models  do  not  offer  robust 
simulations  which  account  for  a  range  of  probability  distributions  while  accounting  for 
risk  across  model  scenarios.  These  products  excel  at  providing  graphical  representations 
of  complex  data  in  the  form  of  digital  “dashboards,”  bubble-charts,  and  efficient 
frontiers.  When  used  in  conjunction  with  PA  modeling  software,  these  PPM  suites  are 
excellent  at  helping  to  efficiently  manage  large  projects  while  helping  to  mitigate  risk. 

The  authors  of  this  paper  found  that  the  Real  Options  Valuation’s  Risk  Simulator 
and  @RISK  were  better  suited  for  the  research  being  conducted  in  accordance  with  the 
intent  of  this  research  paper.  Risk  Simulator  was  by  far  the  superior  product  evaluated 
and  provided  hundreds  of  readymade  models  including;  Military  models,  Efficient 
Frontier  Models,  multiple  simulations.  These  modeling  tools  leverage  the  capabilities  of 
Microsoft  Excel  and  Monte-Carlo  simulation  to  develop  a  range  of  statistical  probability 
distributions  using  an  array  of  variable  inputs.  This  provides  the  ability  to  look  at  the 
best,  worst  and  most  likely  scenarios. 

C.  RESEARCH  LIMITATIONS 

The  authors  found  that  the  “boil  the  ocean”  approach  to  evaluating  products  was 
beyond  the  scope  that  was  executable  in  a  (1)  distributed  team  environment  (i.e.,  team  not 
collocated),  and  (2)  without  a  central  laboratory  environment  with  configuration  control 
of  the  products  being  evaluated.  Additionally,  it  was  difficult  to  codify  all  the  numerous 
requirements  and  variables  from  multiple  sources.  The  model  was  developed  using  only 
a  few  of  the  variables  and  requirements  identified  in  Table  2.  More  complex  models 
could  be  developed  using  the  work  in  this  document  as  a  starting  point. 
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APPENDIX  A.  KEY  REQUIREMENTS  SOURCES 


Table  3.  Key  Requirements  Sources 


Requirements 

Source 

Author 

Key  Requirements 

1.  Dr.  Flynn’s 
research  paper, 
brief,  and  Joint 
Applied  Project 
Prompt 

Dr.  Flynn 

I.  The  3  major  goals  of  portfolio  analysis  as  provided  by 
Dr.  Flynn  are: 

a)  Value  Maximization  (focus  is  ROI) 

b)  Balance  -  Short-term  versus  long-term,  High-risk  versus 
low-risk,  across  product  categories,  Basic  research  vs. 
production 

c)  Strategic  Direction  -  “On-strategy” 

II.  The  major  portfolio  analysis  requirements  (top  level) 
that  have  been  identified  are: 

1.  Baseline  Requirements: 

a)  Gather  Life-Cycle  cost  data 

b)  Establish  Scoring  System  that  determine  how  effectively 
systems  match  capabilities  to  requirements 

c)  Develop  way  to  display  results  to  allow  for  both  risk- 
reward  analysis  and  trade-off  and  establish  a  Scoring 

System 

2.  Process  Requirements  to  follow  during  analysis: 

a)  Maintain  cognizance  of  the  following  inputs  or 
constraints  during  Portfolio  Analysis  (see  spreadsheet  for 
particulars) 

b)  Parameters  to  consider  for  achieving  the  Portfolio 

Analysis  Goal  of  BALANCE  (see  spreadsheet  for 
particulars) 

c)  Requirements  during  the  process  of  Maximizing  Value 
phase  (see  spreadsheet  for  particulars) 

d)  Scorecard  process/contents  requirements  (see  spreadsheet 
for  particulars) 

2.  DoDD  7450.20 
directive 

mandating  use  of 
PA  in  DoD 

systems 

SECDEF 
reference  (c) 

I.  Use  the  Joint  Capability  Area  common  framework 
and  lexicon  for  the  organization  of  capability  portfolios 

II.  Cover  material  and  non-material  investments 

III.  Leverage  operational  expertise  of  Combatant 
Commands 
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Requirements 

Source 


Author 


3.  GAO  Report  on  GAO 
PA  Best  Practices  reference 


Key  Requirements 


Use  of  risk/reward  “bubble  diagram 


(b) 


Figured-:  Risk  Versus  FVwerds-  Miitrii 


4.  Use  of  PA  in 
Logistics  Planning 
Scenarios 


Defense 
Acquisition 
University 
reference  (d) 


Classify  every  purchase  or  family  of  purchases  into  one  of 
four  categories  or  quadrants:  acquisition,  multiple,  leverage, 
or  strategic; 


at 

50 


Acquisition 

Few  suppliers 
Low  value 


Multiple 

Many  suppliers 
Low  value 


Strategic 
Few  suppliers 
High  value 


Leverage 
Many  suppliers 
High  value 


Item  Criticality  or  Value 
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Author 


Requirements 

Source 


Key  Requirements 


5.  Portfolio 


RAND 


Schematic  Depiction  of  Finding  Points  Near  the  Efficient  Frontier 


Analysis  Methods 


reference  (i) 


Efficient  frontier 


6.  Portfolio 


RAND 


RAND  AM6SJJ f.3 


Capabilities  Versus  Time 


Analysis  Methods 


reference  (i) 


Size  of  attack 


Attack  tactics 


Countermeasures 


Geography 


Strategic  warning 


— Near  to  mid-term 
Long  term 

— A —  Very  long  term 


7.  Project 
Management  Body 
of  Knowledge  - 
Functional 
Dimensions 


Project 

Management 

Institute 


I.  Time  management  —  Manages  deliverable  activity 
timelines  and  deadlines  for  programs,  projects,  tasks  and 
assignments 

II.  Resource  management  —  Manages  allocation  of 
available  personnel  using  a  resource  profile  repository,  and 
allows  resource  loading  and  leveling  (natively,  or  at 
minimum  via  third-party  tool  integration) 


III.  Cost  management  —  Tracks  resource  (and,  often, 
other)  costs  and  facilitates  chargeback  or  billing  of  project 
expenses;  for  example,  those  associated  with  time,  travel, 
equipment  or  other  material 
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Requirements 

Source 

Author 

Key  Requirements 

IV.  Integration  (portfolio)  management  —  Provides 
business  intelligence  to  improve  portfolio  alignment, 
planning,  and  project  or  service  delivery.  Dashboard  tools 
provide  an  integrated  view  of  program  and  project  status, 
resource  capacity,  service  levels,  strategic  alignment,  and 
more.  Integration  can  also  be  to  back-office  systems. 

Gathers  portfolio  data  from  these  segments  to  let  users 
manage  their  pipelines,  report  and  forecast  project  progress, 
and  perform  portfolio  analysis  and  prioritization. 

8.  “Assessing 

Risks  and 

Returns:  A  Guide 
for 

Evaluating 

Federal  Agencies’ 
IT  Investment 
Decision-Making,” 

GAO 

reference  (q) 
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9.  “Assessing 

Risks  and 

Returns:  A  Guide 
for 

Evaluating 

Federal  Agencies’ 
IT  Investment 
Decision-Making,” 

GAO 

reference  (q) 

B.  ORGANIZATIONAL  IMPACT  (relative  weight  =  10 
points)  Measures  the  impact  n  organizational  personnel  of 
the  system.  The  more  favorable  the  impact  on  the 
organization  the  higher  its  score. 

B.l.  Personnel  and  Training  (3  of  10  points)  Assess  the 
impact  of  the  system  on  the  knowledge,  skill,  and  training 
of  organizational  personnel  if  the  system  in  implemented. 
Score  from  zero  to  three  based  on  the  scale  below: 

Zero  Points:  System  is  likely  to  require  significant  new 
skills  to  operate  and  support  and  the  project  does  not  appear 
to  mitigate  this  impact  through  appropriate  training,  changes 
in  rating  qualifications,  etc. 

Three  Points:  System  is  an  improvement  to  an  existing 
system  and  will  require  relatively  little  new  skill  and/or 
knowledge  to  operate  or  support.  If  it  is  a  new  system,  it 
will  introduce  valuable  new  skills  and  knowledge  to  the 
organization  and  the  project  will  mitigate  any  adverse 
impact  through  appropriate  training,  planning  for  rating 
qualification  changes,  etc. 
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Author 


Requirements 

Source 


Key  Requirements 


B.2.  Scope  of  Beneficiaries/  Cross-Functional  (4  of  10 

points).  Assess  a  higher  score  (zero  to  four)  the  broader  the 
scope  of  beneficiaries. 

Zero  Points:  Limited  number  of  beneficiaries.  This  system 
will  be  used  by  only  one  office  in  headquarters,  a  single 
area,  or  district.  Not  a  cross-functional  system. 

Four  Points:  System  is  cross-functional  and  serves  a  number 
of  offices,  areas,  and/or  districts.  System  will  be  used  by  a 
large  number  of  organizational  units.  System  will  be  used 
by  the  public. 

B. 3.  Quality  of  Work  Life  (3  of  10  points).  Measures  the 
improvement  in  quality  of  work  life  expected  for  the 
systems.  Score  higher  (zero  to  three  points)  the  more  work 
life  improvement  is  expected. 

Zero  Points:  Little  if  any  positive  impact  on  the  quality  of 
work  life.  System  may  increase  the  work  required  (e.g., 
additional  data  entry). 

Three  Points:  Positive  contribution  to  the  quality  of  work 
life  will  clearly  result.  For  example,  the  system  will  improve 
medical  care  for  dependents  or  allow  a  job  to  be  done  much 
faster  such  that  job  satisfaction  will  increase. 

C.  MISSION  EFFECTIVENESS  (relative  weight  =  20 
points).  Measures  the  impact  of  the  system  on  both  external 
and  internal  customers.  It  is  a  measure  of  the  system’s 
ability  to  improve  the  performance  of  support  or  operational 
programs.  This  improvement  should  be  measured  in 
quantitative  terms,  but  not  in  dollars.  The  economic  (dollar) 
impact  is  captured  in  the  benefit/  cost  ratio.  However,  the 
same  benefits  might  be  measured  here  in  a  different  manner. 
For  example,  improvements  might  be  expressed  in  terms  of 
accomplishing  a  task  sooner  (hours  or  minutes),  delivering  a 
service  with  fewer  mistakes,  increasing  the  availability  of  a 
computer  system  for  customer  use  (hours  per  month  saved 
in  time  for  system  backups),  or  a  number  of  similar  terms. 
The  more  the  project  or  system  improves  mission 
effectiveness  the  higher  the  score. 

C.l.  Improve  Internal  Program  Services  (10  of  20 

points).  Assess  the  expected  improvement  in  service  to 
internal  customers.  For  example  the  system  might  improve 
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Author 


Requirements 

Source 


Key  Requirements 


the  timeliness  of  financial  reporting  throughout  the 
organization.  Score  (zero  to  ten)  higher,  the  more  that 
service  will  be  improved  in  response  to  a  problem  expressed 
by  users  of  the  service. 


Zero  Points:  System  does  not  appear  to  meet  a  problem 
defined  by  an  internal  customer.  Little  improvement  in 
important  customer  service  criteria,  such  as  timeliness, 
quality,  or  availability  is  expected.  An  improvement  is 
described  but  not  quantified. 


Ten  Points:  A  significant  improvement  expected  in  areas 
such  as  timeliness,  quality,  or  availability  and  the 
improvement  is  quantified.  The  improvement  also  addresses 
an  important  problem  or  area  of  service  improvement 
defined  by  the  customer. 


C.2.  Improved  Service  to  the  Public  (10  of  20  points). 
Assess  the  expected  improvement  in  service  to  the  public. 
Score  (zero  to  ten)  higher,  the  more  improvement  is 
anticipated  in  response  to  a  requirement  defined  by  the 
public. 


Zero  Points:  System  appears  to  provide  little  or  no  direct 
improvement  in  service  to  the  public.  Systems  may  make  a 
small  improvement  in  timeliness,  quality,  or  availability,  but 
there  is  no  documented  need  for  such  an  improvement.  The 
improvement  is  not  quantified. 


Ten  Points:  System  significantly  improves  service  to  the 
public  in  a  mission  where  need  is  demonstrated  or  provides 
a  new  type  of  service  to  meet  changing  customer  demands. 
The  improvement  is  quantified. 


D.  STRATEGIC  ALIGNMENT  (relative  weight  =  25 
points).  Measures  to  what  extent  the  proposed  investment 
supports  strategic  organizational  objectives.  Scoring  is 
based  primarily  on  explicit  documentation  of  the  need  for 
the  Integrated  Risk  Management  (IRM)  system  in  planning 
documents.  The  more  the  project  or  system  is  aligned  with 
program/strategic  goals,  the  higher  the  score. 

D.l.  Business  Model  (7  of  25  points).  Assess  the  degree  of 
alignment  with  the  organization’s  business  model. 
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Author 


Requirements 

Source 


Key  Requirements 


Zero  Points:  Proposed  project  or  system  does  not  support 
organizational  products/services  or  processes  identified  in 
the  business  model. 


One  to  Four  Points:  Proposed  system  is  specifically 
mentioned  in  the  5 -year  IRM  plan  and  supports 
organizational  products/services  or  processes  identified  in 
the  business  model.  (Score  one  to  two  points  if  the  system 
supports  products/services  or  processes  in  the  business 
model,  but  is  not  listed  in  the  5-year  IRM  plan.) 

Five  to  Seven  Points:  Proposed  system  is  specifically 
mentioned  in  the  5 -year  IRM  plan  and  supports 
products/services  or  processes  identified  in  the  business 
model,  and  the  project  has  been  coordinated  with  all  offices 
identified  by  the  model  for  the  respective  processes  the 
system  supports. 

D.3.  Business  Process  Redesign  (6  of  25  points).  Assess 
the  degree  this  system  enables  the  organization  to  do 
business  in  a  better  way.  Score  (zero  to  six)  higher  the 
greater  the  expected  improvement. 

Zero  Points:  This  system  automates  an  existing  business 
process  with  little  improvement  of  the  process  (i.e.,  helps  to 
do  the  same  thing  faster). 

Six  Points:  System  enables  a  significant  improvement  in  the 
way  business  is  conducted 

D. 2.  Level  of  Interest  (12  of  25  points).  Assess  the  level  of 
interest  by  senior  managers  (at  agency  and  departmental 
level)  and/or  the  Congress.  Score  (zero  to  twelve)  higher  the 
greater  the  level  of  interest. 

Zero  Points:  No  expressed  support  for  this  system  by  senior 
managers  or  the  Congress. 

Twelve  Points:  System  has  strong  support  from  the 
Congress,  departmental  senior  managers,  and/or  the  head  of 
the  agency.  System  is  specifically  mentioned  in 
determinations. 

E.  BENEFIT-COST  IMPACT(S)  (relative  weight  =  25 
points).  Measures  the  value  of  the  system  in  dollar  terms. 
The  system  benefit/cost  ration  is  the  key  indicator.  This 
ration  is  developed  using  the  standard  benefit-cost  guidance 
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Author 


Requirements 

Source 


Key  Requirements 


and  spreadsheet  promulgated  by  the  organization.  The 
standard  guidance  ensures  all  system  studies  include  a 
common  set  of  costs  and  approach  benefits  definition  in  a 
similar  manner.  The  standard  spreadsheet  assists  system 
sponsors  in  the  benefit/cost  calculation.  The  higher  the 
benefit/cost  ratio,  the  better  the  score. 

Zero  Points:  Any  benefit/cost  ratio  less  than  one  (i.e.,  costs 
exceed  the  benefits). 

One  Point:  Benefit/cost  ratio  of  one. 

Five  Points:  Benefit/cost  ratio  of  1.5  to  1.75. 

Ten  Points:  Benefit/cost  ratio  of  1.76  to  1.99. 

Fifteen  Points:  Benefit/cost  ratio  of  2.0  to  2.99. 

Twenty  Points:  Benefit/cost  ratio  of  3.0  to  3.99. 

Twenty-five  Points:  Benefit/cost  ratio  of  4.0  or  greater 
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APPENDIX  B.  PALISADE  @RISK  MODELING  DATA 

AND  ANALYSIS 


A.  PALISADE  @RISK  MODELING  DATA 

In  this  model,  the  user  provides  the  Project  Name,  the  Budget  for  the  project,  and 
then  provides  Minimum,  Most  Likely  (ML)  and  Maximum  budget  values.  The  individual 
project  percentages  should  reflect  of  the  actual  cost  associated  with  the  project  as 
represented  by  the  ML  and  Maximum  budget  overruns  expected.  This  model  could  be 
enhanced  with  additional  variables  representing  project  risk,  or  ranges  of  risk  represented 
as  Min/ML/Max  values.  The  user  may  change  any  of  the  light  green  shaded  cells,  which 
are  the  model’s  inputs;  the  significant  model  outputs  are  shaded  in  light  blue. 

1.  @RISK  Model  using  Minimum,  Most  Likely  (ML)  and  Maximum 
Budget  Values 


Table  4.  Budget  Simulation 


Project 

Budget 

Min 

ML 

Max 

Min$ 

ML$ 

Max$ 

Budget 

Simulation 

1000  Iterations 

Project  1 

$1,732.44 

100% 

110% 

125% 

$ 

1,732.44 

$ 

1,905.68 

$  2,165.55 

$ 

1,920.12 

Project  2 

$859.00 

100% 

110% 

125% 

$ 

859.00 

$ 

944.90 

$ 

1,073.75 

$ 

952.06 

Project  3 

$1,845.00 

100% 

110% 

125% 

$ 

1,845.00 

$ 

2,029.50 

$ 

2,306.25 

$ 

2,044.88 

Project  4 

$1,645.00 

100% 

110% 

125% 

$ 

1,645.00 

$ 

1,809.50 

$ 

2,056.25 

$ 

1,823.21 

Project  5 

$458.00 

100% 

110% 

125% 

$ 

458.00 

$ 

503.80 

$ 

572.50 

$ 

507.62 

Project  6 

$52.00 

100% 

110% 

125% 

$ 

52.00 

$ 

57.20 

$ 

65.00 

$ 

57.63 

Project  7 

$758.00 

100% 

110% 

125% 

$ 

758.00 

$ 

833.80 

$ 

947.50 

$ 

840.12 

Project  8 

$115.00 

100% 

110% 

125% 

$ 

115.00 

$ 

126.50 

$ 

143.75 

$ 

127.46 

Project  9 

$125.00 

100% 

110% 

125% 

$ 

125.00 

$ 

137.50 

$ 

156.25 

$ 

138.54 

Project  10 

$458.00 

100% 

110% 

125% 

$ 

458.00 

$ 

503.80 

$ 

572.50 

$ 

507.62 

Project  1 1 

$45.00 

100% 

110% 

125% 

$ 

45.00 

$ 

49.50 

$ 

56.25 

$ 

49.88 

Project  12 

$105.00 

100% 

110% 

125% 

$ 

105.00 

$ 

115.50 

$ 

131.25 

$ 

116.38 

Project  13 

$48.00 

100% 

110% 

125% 

$ 

48.00 

$ 

52.80 

$ 

60.00 

$ 

53.20 

Project  14 

$351.00 

100% 

110% 

125% 

$ 

351.00 

$ 

386.10 

$ 

438.75 

$ 

389.03 

Project  15 

$421.00 

100% 

110% 

125% 

$ 

421.00 

$ 

463.10 

$ 

526.25 

$ 

466.61 

Project  16 

$124.00 

100% 

110% 

125% 

$ 

124.00 

$ 

136.40 

$ 

155.00 

$ 

137.43 

Project  17 

$521.00 

100% 

110% 

125% 

$ 

521.00 

$ 

573.10 

$ 

651.25 

$ 

577.44 
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Project 

Budget 

Min 

ML 

Max 

Min$ 

ML$ 

Max$ 

Budget 

Simulation 

1000  Iterations 

Project  18 

$512.00 

100% 

110% 

125% 

$ 

512.00 

$ 

563.20 

$ 

640.00 

$ 

567.47 

Project  19 

$5.00 

100% 

110% 

125% 

$ 

5.00 

$ 

5.50 

$ 

6.25 

$ 

5.54 

Project  20 

$21.00 

100% 

110% 

125% 

$ 

21.00 

$ 

23.10 

$ 

26.25 

$ 

23.28 

Total 

$10,200.44 

$ 

10,200.44 

$ 

11,220.48 

$  12,750.55 

$ 

11,305.48 

The  distribution  curve  for  the  budget  simulation  is  shown  in  Table  4  above.  This 
model  uses  the  previous  results  from  1000  simulations  to  arrive  at  a  90%  probability  that 
the  projects  can  be  executed  at  a  budget  cost  (Simulated  Budget)  of  $11,514,  or  a  delta 
cost  increase  of  $1,314  from  the  budget  baseline  of  $10,200  (Figure  41). 
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Figure  4 1 .  Cost  Probability  Distribution 
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Building  upon  the  results  of  the  previous  model,  in  the  next  model,  Table  5,  the 
Project  Name  and  the  Mean  Budget  Simulation  are  inherited  from  the  previous  model. 
New  columns  for  Strategic  Value,  Tactical  Value  and  FTE  Resources  are  added.  These 
columns  are  scores  (which  would  be)  provided  by  a  panel  of  military  experts  which  can  be 
considered  a  Delphi  panel;  for  this  model,  the  scores  were  determined  by  the  authors.  The 
simple  addition  of  these  the  Delphi  scores  are  added  to  create  an  overall  KVA  score.  The 
EMV  is  then  calculated  by  dividing  the  KVA  with  the  simulated  budget  value.  The  EMV 
is  now  akin  to  a  ROI  which  is  used  to  evaluate  and  rank  commercial  projects.  Note  that 
the  Value  scores  are  not  weighted  in  this  model. 

2.  @RISK  Model  using  Strategic,  Tactical  and  FTE  Resource  values 


Table  5.  Expected  Military  Value 


Projects 

Budget 

Simulation 

Strategic 

Value 

Tactical 

Value 

FTE 

Resources 

KVA 

Scores 

EMV 

(KVA/Budget) 

11.61 

0.605% 

8.60 

0.903% 

18.23 

0.892% 

14.94 

0.819% 

16.77 

3.303% 

18.63 

32.333% 

24.24 

2.885% 

22.27 

17.469% 

14.37 

10.371% 

17.20 

3.389% 

17.07 

34.216% 

15.83 

13.603% 

9.28 

17.440% 

28.80 

7.403% 

23.73 

5.085% 

24.55 

17.860% 

12.35 

2.138% 

10.29 

1.813% 

10.57 

190.741% 

11.27 

48.421% 

Total 

$  11,305.48 
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In  this  model,  the  values  are  inherited  from  the  previous  two  models.  The  model  is 
then  sorted  in  descending  sequence  by  EMV. 

3.  @RISK  Model  Sorted  by  EMV 

Table  6  below  shows  the  top  ranked  EMV.  The  key  aspects  for  the  models  are  the 
expected  budget  values,  and  the  scores  provided  by  the  Delphi  panel  of  experts.  The 
validity  of  the  input  values  is  critical  to  obtaining  usable  results  from  these  @RISK 
models. 


Table  6.  Top  Ranked  EMV 


Programs 

Budget  Simulation 

KVA 

Scores 

EMV 

(KVA/Budget) 

Project  19 

$5.54 

10.57 

190.741% 

Project  20 

$23.28 

11.27 

48.421% 

Project  11 

$49.88 

17.07 

34.216% 

Project  6 

$57.63 

18.63 

32.333% 

Project  16 

$137.43 

24.55 

17.860% 

Project  8 

$127.46 

22.27 

17.469% 

Project  13 

$53.20 

9.28 

17.440% 

Project  12 

$116.38 

15.83 

13.603% 

Project  9 

$138.54 

14.37 

10.371% 

Project  14 

$389.03 

28.80 

7.403% 

Project  15 

$466.61 

23.73 

5.085% 

Project  10 

$507.62 

17.20 

3.389% 

Project  5 

$507.62 

16.77 

3.303% 

Project  7 

$840.12 

24.24 

2.885% 

Project  17 

$577.44 

12.35 

2.138% 

Project  18 

$567.47 

10.29 

1.813% 

Project  2 

$952.06 

8.60 

0.903% 

Project  3 

$2,044.88 

18.23 

0.892% 

Project  4 

$1,823.21 

14.94 

0.819% 

Project  1 

$1,920.12 

11.61 

0.605% 

Total 

$  11,305.48 
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4. 


Palisade  @RISK  Data  Analysis 


This  section  provides  graphical  representations  of  the  preceding  data  tables,  and 
provides  analysis  of  the  information.  The  graphics  in  Figure  42  and  Figure  43  compare 
the  original  budget  values  with  the  simulated  budget  values  following  1,000  Monte  Carlo 
simulation  iterations.  The  delta  between  the  budgets  (original  vs.  simulated)  is  driven  by 
the  low,  most-likely  and  maximum  cost  values  provided  to  the  model. 
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Figure  42.  Budget  (Original  vs.  Simulated  #1) 


The  graphic  in  Figure  44  compares  the  project  cost  (Sim$)  with  the  EMV  for  each 
project.  Projects  where  the  EMV  point  is  above  the  Sim$  point  are  the  most  favorable 
from  an  EMV  perspective.  In  this  particular  simulation,  the  highest  cost  projects  appear  to 
have  the  least  return  (based  upon  the  notional  weighting  applied  to  the  KVA). 
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Figure  43.  Budget  (Original  vs.  Simulated  #2) 
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Figure  44.  Project  Cost  (Sim$)  Compared  with  EMV 
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The  graphic  in  Figure  45  stratifies  the  EMV  low  to  high.  In  this  simulation,  only 
(4)  projects  (#6,  1 1,  20,  19)  have  an  EMY  greater  than  20%. 
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Figure  45.  Project  EMV  Low  to  High 


While  this  model  seems  to  provide  for  a  solution  to  the  original  model,  it  in  fact 
does  not.  The  different  models  are  not  integrated  with  each  other  which  does  not  allow  for 
a  continuous  simulation,  optimization  cycle  that  is  required  to  create  a  valid  efficient 
frontier.  This  would  be  one  of  the  software  programs  used  by  the  NPS  PA  CoE  to 
evaluate  the  possibility  of  being  utilized  for  future  implementation,  though  at  this  time,  it 
is  not  a  viable  candidate  for  evaluating  defense  acquisition  capability  requirements. 
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APPENDIX  C.  CREATION  OF  AN  EFFICIENT  FRONTIER 


Using  Risk  Simulator  to  create  an  efficient  frontier  is  easily  accomplished.  As 
depicted  in  Figure  46,  select  any  green  cell  (A)  in  the  model  (these  are  already  set  up 
Monte  Carlo  simulation  assumptions).  Reset  or  set  new  assumptions  by  clicking  on  Risk 
Simulator,  Set  Input  Assumptions  (B)  and  selecting  the  desired  distribution.  In  this  case, 
select  Triangular  (C)  and  enter  in  the  desired  Minimum,  Likely,  and  Maximum  values 
(C). 


Figure  46.  Efficient  Frontier  Setting  Assumptions 

Next,  select  the  desired  output.  In  the  example  depicted  in  Figure  47,  cell  C26  (D) 
is  the  total  NPV  for  the  portfolio  (EMV  can  also  be  selected  if  desired).  Click  on  Set 
Output  Forecast  (E)  and  give  provide  a  name  (F).  Run  Simulation  or  Run  Super  Speed 
(G),  which  displays  chart  (H).  Set  the  probability  of  exceeding  or  being  below  some 
value  (H).  You  can  then  analyze  the  Statistics  tab  to  look  at  all  the  risk  factors  and  risk 
coefficients  (e.g.,  coefficient  of  variation).  You  can  also  see  a  two  tail  or  left  tail 
boundary,  as  well  as  adjust  the  percent  of  certainty  in  the  chart. 
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Figure  47.  Setting  the  Output 

Now  you  are  ready  to  run  the  optimization  (I)  in  Figure  48.  Here  you  can  select 
the  Static,  Dynamic  or  Stochastic  Optimization  routine  (K).  As  shown  in  (K)  below, 
simulation  is  tied  into  the  optimization  routines  (i.e.,  these  two  items  cannot  be  separated 
into  two  different  applications).  When  optimizing  using  either  the  Dynamic  or  Stochastic 
routines,  you  will  need  to  simulate,  optimize,  and  then  repeat  multiple  times. 


Figure  48.  Starting  Optimization 
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As  shown  in  (L)  below  in  Figure  49  you  can  also  set  constraints,  then  run  the 
optimization  model  (M)  again  to  add  in  the  Efficient  Frontier  (N)  by  selecting  the 
constraints  of  interest  (O)  and  adding  them  to  the  changing  constraints  list  to  run  the 
efficient  frontier. 
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Figure  49.  Setting  Constraints 


The  efficient  frontiers  generated  by  the  optimization  illustrate  the  best 
combination  and  permutation  of  projects  in  the  optimal  portfolio.  Each  point  on  the 
frontier  is  a  portfolio  of  various  combinations  of  projects  that  provide  the  best  allocation 
possible  given  the  requirements  and  constraints  (Mun  &  Housel,  2006,  p.  12).  Because 
this  is  an  integrated  process,  simulated  and  optimized  in  sequence,  together,  the  analysis 
will  also  be  integrated,  which  is  why  it  is  called  Integrated  Risk  Management  (IRM). 
Since  the  process  is  integrated,  it  cannot  be  taken  apart  to  allow  some  parts  to  be  done 
one  way  and  other  parts  to  be  done  in  another  way  and  then  merged  back  together.  If  the 
analysis  is  performed  in  a  way  that  is  not  integrated,  the  results  will  be  erroneous. 
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